Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 15.483666753s trace: https://jaeger.dev.storj.io/trace/1935d16c4ec3317c FAIL (2025-12-29 05:08:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 6:41.757908043 (565h37m41.628991078s) [noise=true , address=64.49.13.27:4959 , piece_id=6IHDQ6H64TI5Q653TPSGUTEZQYNABZCZO7AU26R2OLARLCUEPNZQ, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.758607134 (565h37m41.629690189s) [noise=true , address=145.79.227.44:4220 , piece_id=5UMM6PIETJ2M5QLGNZNPQ6BZCC4QWG4HUX4YTFHX2ATAQJICH53A, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.758997888 (565h37m41.630080933s) [noise=true , address=45.91.116.11:4124 , piece_id=FLAGSPMHPDJPGMBLHYBPDJKBWVKEQVEEZWVWZVAQQ6UHR6UYPEBA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.759288822 (565h37m41.630371867s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.759638187 (565h37m41.630721222s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760079808 (565h37m41.631162853s) [noise=true , address=145.79.227.49:4357 , piece_id=KXH4ZP22GRTNM2PUG56YHTQDSSOZTNLF4HZJ3ASPWUVSGK3U4SMA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760363528 (565h37m41.631446573s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760702464 (565h37m41.631785509s) [noise=true , address=145.79.227.12:4789 , piece_id=SLLU5XYDUV3TY7ETYJWXJHEZKA4RMLM5YVVEZG7YVPIVFGMYLL7Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.761056638 (565h37m41.632139673s) [noise=true , address=145.79.227.29:4975 , piece_id=TCVWZ65DXRLO4XXS4QURGXKGRFYXHQQSQY634FVU6TSGMTVC6IZQ, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.767898968 (565h37m41.638982013s) [noise=true , address=77.90.31.10:4525 , piece_id=CNBB6FZFANMK46XOWGIBIUNKF725QRRSIZPWIM2JRRX4RGMSHYIA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.769576983 (565h37m41.640660018s) [noise=true , address=145.79.227.14:4852 , piece_id=WDSJH7ALDJJLPTAANMI2E7JSLD7PBYKTJ32JQY7ILVORZUS6X7JA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.771126823 (565h37m41.642209888s) [noise=true , address=64.49.13.140:4514 , piece_id=VB4NOBKANE3OMUUURFJLSW6XZKVX32X4JTZRVKKTI26V7ZIDGR3Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.772734214 (565h37m41.643817259s) [noise=true , address=145.79.227.30:4796 , piece_id=OQGVOFUF6DY3Q3ZAOSOKLFLH47L4YMSW2SGOIFXYQ3TCXZLTJE6Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.774688165 (565h37m41.64577121s) [noise=true , address=77.90.31.13:4653 , piece_id=NGNJ2K55GTD3OFTYLDCUIT33OC5ZZ6QJYPF53HN62XXXKNGE64LA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.776343948 (565h37m41.647426983s) [noise=true , address=77.90.31.18:4830 , piece_id=MEJY5LQTTJV3R3MJSALXWU4THTD3XNZJJZ3Y7TQQOTIYMZBODMUA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.777891275 (565h37m41.64897431s) [noise=true , address=145.79.227.40:4072 , piece_id=SSWX7TAWXQKWBTFPBQR7FYQZFZVANVVVCVQJ45J46DS6XMGGEWCA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.779496952 (565h37m41.650579997s) [noise=true , address=145.79.227.4:4209 , piece_id=GYRCBERNVCY6BWYRRXK2GVHRCLFVXZIHD4JWBEAHNK3MKHDFDETA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:42.700027515 (565h37m42.57111055s) [noise=true , address=64.49.13.27:4959 , piece_id=6IHDQ6H64TI5Q653TPSGUTEZQYNABZCZO7AU26R2OLARLCUEPNZQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: ecclient: upload failed (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy, address:64.49.13.27:4959): protocol: storage node overloaded, request limit: 2000 Dec 29 05:06:43.064378199 (565h37m42.935461234s) [upload=495917]: Next returned nil segment Dec 29 05:06:43.064393940 (565h37m42.935476975s) [upload=495917]: Waiting for error group managing segments... Dec 29 05:06:43.492698353 (565h37m43.363781388s) [noise=true , address=64.49.13.140:4514 , piece_id=VB4NOBKANE3OMUUURFJLSW6XZKVX32X4JTZRVKKTI26V7ZIDGR3Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.620875843 (565h37m43.491958879s) [noise=true , address=145.79.227.12:4789 , piece_id=SLLU5XYDUV3TY7ETYJWXJHEZKA4RMLM5YVVEZG7YVPIVFGMYLL7Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.623709580 (565h37m43.494792635s) [noise=true , address=145.79.227.14:4852 , piece_id=WDSJH7ALDJJLPTAANMI2E7JSLD7PBYKTJ32JQY7ILVORZUS6X7JA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.634947790 (565h37m43.506030835s) [noise=true , address=145.79.227.4:4209 , piece_id=GYRCBERNVCY6BWYRRXK2GVHRCLFVXZIHD4JWBEAHNK3MKHDFDETA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.660936426 (565h37m43.532019461s) [noise=true , address=145.79.227.29:4975 , piece_id=TCVWZ65DXRLO4XXS4QURGXKGRFYXHQQSQY634FVU6TSGMTVC6IZQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.860684970 (565h37m43.731768016s) [noise=true , address=77.90.31.10:4525 , piece_id=CNBB6FZFANMK46XOWGIBIUNKF725QRRSIZPWIM2JRRX4RGMSHYIA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.897098368 (565h37m43.768181413s) [noise=true , address=77.90.31.13:4653 , piece_id=NGNJ2K55GTD3OFTYLDCUIT33OC5ZZ6QJYPF53HN62XXXKNGE64LA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.928905363 (565h37m43.799988408s) [noise=true , address=77.90.31.18:4830 , piece_id=MEJY5LQTTJV3R3MJSALXWU4THTD3XNZJJZ3Y7TQQOTIYMZBODMUA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.971303356 (565h37m43.842386401s) [noise=true , address=45.91.116.11:4124 , piece_id=FLAGSPMHPDJPGMBLHYBPDJKBWVKEQVEEZWVWZVAQQ6UHR6UYPEBA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.541477983 (565h37m44.412561018s) [noise=true , address=145.79.227.49:4357 , piece_id=KXH4ZP22GRTNM2PUG56YHTQDSSOZTNLF4HZJ3ASPWUVSGK3U4SMA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.554957591 (565h37m44.426040626s) [noise=true , address=145.79.227.43:4191 , piece_id=4GRQACHDAIEOXTFLCAQXS2ORNXMTOU6VSTE7YBAITEXRUWAO5JYQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.570831037 (565h37m44.441914072s) [noise=true , address=145.79.227.44:4220 , piece_id=5UMM6PIETJ2M5QLGNZNPQ6BZCC4QWG4HUX4YTFHX2ATAQJICH53A, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.580290328 (565h37m44.451373363s) [noise=true , address=145.79.227.40:4072 , piece_id=SSWX7TAWXQKWBTFPBQR7FYQZFZVANVVVCVQJ45J46DS6XMGGEWCA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.655455876 (565h37m44.526538911s) [noise=true , address=145.79.227.30:4796 , piece_id=OQGVOFUF6DY3Q3ZAOSOKLFLH47L4YMSW2SGOIFXYQ3TCXZLTJE6Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.655480864 (565h37m44.526563909s) [seg_pos={0 0}, upload=495917]: Segment reached optimal threshold of 14 pieces. Dec 29 05:06:44.655678942 (565h37m44.526761977s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:44.655686626 (565h37m44.526769671s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.655850348 (565h37m44.526933383s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection Dec 29 05:06:44.655857702 (565h37m44.526940747s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.656053324 (565h37m44.527136369s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:44.656061891 (565h37m44.527144926s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.656092720 (565h37m44.527175765s) [seg_pos={0 0}, upload=495917]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:06:44.656214661 (565h37m44.527297696s) [upload=495917]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:06:44.656221103 (565h37m44.527304149s) [upload=495917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:06:44.975569917 (565h37m44.846652962s) [upload=495917]: Done uploading segments... Dec 29 05:06:44.975596969 (565h37m44.846680004s) [upload=495917]: Upload finished. err: <nil> Duration: 15.483694475s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.....XX.......X.XX..XX..XXX........XX......X...X..XXX..X..] duration: 5.708623929s trace: https://jaeger.dev.storj.io/trace/166375b545182e5f FAIL (2025-12-29 05:08:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q) context deadline exceeded ======LOGS====== 29 05:06:36.970550676 (565h37m36.841633711s) [upload=495915]: Starting upload Dec 29 05:06:36.970689340 (565h37m36.841772385s) [upload=495915]: Done starting upload Dec 29 05:06:36.970766808 (565h37m36.841849843s) [upload=495915]: Uploading segments... Dec 29 05:06:36.970956749 (565h37m36.842039784s) [upload=495915]: Got next segment. Inline: false Dec 29 05:06:36.971001284 (565h37m36.842084319s) [upload=495915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:06:36.971007365 (565h37m36.8420904s) [upload=495915]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:06:37.219391530 (565h37m37.090474565s) [seg_pos={0 0}, upload=495915]: Begin upload segment... Dec 29 05:06:37.219442347 (565h37m37.090525382s) [seg_pos={0 0}, upload=495915]: Done begin upload segment. Dec 29 05:06:37.219473056 (565h37m37.090556091s) [noise=true , address=77.90.31.12:4589 , piece_id=LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.219523822 (565h37m37.090606867s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220113205 (565h37m37.09119625s) [noise=true , address=77.90.31.14:4663 , piece_id=ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220540107 (565h37m37.091623142s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220551049 (565h37m37.091634094s) [noise=true , address=145.79.227.29:4942 , piece_id=ZG3PNUCUXGUA2QP4LEWPQS7GOYYPAHXAKXMZDNKOJCP2BWS5KHKQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220947343 (565h37m37.092030378s) [noise=true , address=145.79.227.50:4374 , piece_id=HSIGXR74IPP6SOW7LGVWUWPOG3ALODIE5MM7BYNCX7VZGUHLJHCA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221017587 (565h37m37.092100632s) [noise=true , address=145.79.227.13:4804 , piece_id=SX6GJDLYSEIO2VMC3CZ3QXE5ED74AOYQFWB6ILJ6HBXTVODUBQZA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221232225 (565h37m37.09231526s) [noise=true , address=77.90.31.11:4555 , piece_id=O4PLT2VBZD3MHESP6BNYD4P6KPB3TWQO7INKBQ3K7VXUVSU2OYXA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221437186 (565h37m37.092520231s) [noise=true , address=145.79.227.46:4602 , piece_id=CMVURWNPAEJFFW7OLNI3LGT5ERT2CVHBAJ32WRF5NRC5KUEWPUJA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221512349 (565h37m37.092595394s) [noise=true , address=45.91.116.4:4937 , piece_id=47MMITQEIU2ROUVT3DGLXUKW2KQTYQKTL3B2JXTKISBGAJVGFGYQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221787312 (565h37m37.092870347s) [noise=true , address=45.91.116.13:4156 , piece_id=NFXHBJZRTUTLPCCOSIVHPFQEL54ZLRW7VRYGJ3RGEPUHMQYAF73Q, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222126538 (565h37m37.093209574s) [noise=true , address=145.79.227.42:4123 , piece_id=C3J47FCY5NHBU2EVOYJWL2TXIZM3T2NP4QJR7DCXA5OE66I2MTJA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222409818 (565h37m37.093492863s) [noise=true , address=145.79.227.47:4266 , piece_id=5EIC6KID2TXVUISCFF2JGVP2ROEQWBFDVWIZMJ4PLYO2E4ZQ4VFA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222766296 (565h37m37.093849331s) [noise=true , address=145.79.227.24:4582 , piece_id=BYRUYICBLZPAWQMT3VH6OPDOW2M7T7RT2BQ7KCYVML2T3X77UGQQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.223062421 (565h37m37.094145466s) [noise=true , address=145.79.227.52:4446 , piece_id=G2AOIQ6NGVCGZ5HVFWUBJC7FCSDGRZ3QUVWT5Y2XCLF3BLC52EFQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.223388491 (565h37m37.094471526s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:38.912022873 (565h37m38.783105908s) [upload=495915]: Next returned nil segment Dec 29 05:06:38.912038503 (565h37m38.783121538s) [upload=495915]: Waiting for error group managing segments... Dec 29 05:06:39.196335632 (565h37m39.067418667s) [noise=true , address=145.79.227.13:4804 , piece_id=SX6GJDLYSEIO2VMC3CZ3QXE5ED74AOYQFWB6ILJ6HBXTVODUBQZA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.211162354 (565h37m39.08224539s) [noise=true , address=145.79.227.52:4446 , piece_id=G2AOIQ6NGVCGZ5HVFWUBJC7FCSDGRZ3QUVWT5Y2XCLF3BLC52EFQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.212495473 (565h37m39.083578508s) [noise=true , address=145.79.227.42:4123 , piece_id=C3J47FCY5NHBU2EVOYJWL2TXIZM3T2NP4QJR7DCXA5OE66I2MTJA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.213852737 (565h37m39.084935772s) [noise=true , address=145.79.227.50:4374 , piece_id=HSIGXR74IPP6SOW7LGVWUWPOG3ALODIE5MM7BYNCX7VZGUHLJHCA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.218895510 (565h37m39.089978545s) [noise=true , address=145.79.227.47:4266 , piece_id=5EIC6KID2TXVUISCFF2JGVP2ROEQWBFDVWIZMJ4PLYO2E4ZQ4VFA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.226304798 (565h37m39.097387843s) [noise=true , address=145.79.227.24:4582 , piece_id=BYRUYICBLZPAWQMT3VH6OPDOW2M7T7RT2BQ7KCYVML2T3X77UGQQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.227391057 (565h37m39.098474102s) [noise=true , address=145.79.227.46:4602 , piece_id=CMVURWNPAEJFFW7OLNI3LGT5ERT2CVHBAJ32WRF5NRC5KUEWPUJA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.360080590 (565h37m39.231163625s) [noise=true , address=145.79.227.29:4942 , piece_id=ZG3PNUCUXGUA2QP4LEWPQS7GOYYPAHXAKXMZDNKOJCP2BWS5KHKQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.402523819 (565h37m39.273606854s) [noise=true , address=77.90.31.14:4663 , piece_id=ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.423611764 (565h37m39.294694809s) [noise=true , address=77.90.31.11:4555 , piece_id=O4PLT2VBZD3MHESP6BNYD4P6KPB3TWQO7INKBQ3K7VXUVSU2OYXA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.492584129 (565h37m39.363667164s) [noise=true , address=45.91.116.4:4937 , piece_id=47MMITQEIU2ROUVT3DGLXUKW2KQTYQKTL3B2JXTKISBGAJVGFGYQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.500637974 (565h37m39.371721009s) [noise=true , address=45.91.116.13:4156 , piece_id=NFXHBJZRTUTLPCCOSIVHPFQEL54ZLRW7VRYGJ3RGEPUHMQYAF73Q, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:40.183710850 (565h37m40.054793885s) [noise=true , address=77.90.31.12:4589 , piece_id=LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:40.183730306 (565h37m40.054813351s) [seg_pos={0 0}, upload=495915]: Segment reached optimal threshold of 13 pieces. Dec 29 05:06:40.183885813 (565h37m40.054968848s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:40.183892525 (565h37m40.05497557s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184152319 (565h37m40.055235355s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection Dec 29 05:06:40.184158902 (565h37m40.055241947s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184268852 (565h37m40.055351897s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:40.184274713 (565h37m40.055357748s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184290533 (565h37m40.055373578s) [seg_pos={0 0}, upload=495915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:06:40.184360165 (565h37m40.05544321s) [upload=495915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:06:40.184365886 (565h37m40.055448931s) [upload=495915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:06:40.481038220 (565h37m40.352121275s) [upload=495915]: Done uploading segments... Dec 29 05:06:40.481063508 (565h37m40.352146553s) [upload=495915]: Upload finished. err: <nil> Duration: 5.708653916s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....X.....X..XX......X.X.....X.....X........X..X...X....X...] duration: 3.527168515s trace: https://jaeger.dev.storj.io/trace/2cf94efd24994e89 FAIL (2025-12-29 05:08:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527204344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 34.730359089s trace: https://jaeger.dev.storj.io/trace/1935d16c4ec3317c FAIL (2025-12-29 05:08:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 6:41.757908043 (565h37m41.628991078s) [noise=true , address=64.49.13.27:4959 , piece_id=6IHDQ6H64TI5Q653TPSGUTEZQYNABZCZO7AU26R2OLARLCUEPNZQ, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.758607134 (565h37m41.629690189s) [noise=true , address=145.79.227.44:4220 , piece_id=5UMM6PIETJ2M5QLGNZNPQ6BZCC4QWG4HUX4YTFHX2ATAQJICH53A, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.758997888 (565h37m41.630080933s) [noise=true , address=45.91.116.11:4124 , piece_id=FLAGSPMHPDJPGMBLHYBPDJKBWVKEQVEEZWVWZVAQQ6UHR6UYPEBA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.759288822 (565h37m41.630371867s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.759638187 (565h37m41.630721222s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760079808 (565h37m41.631162853s) [noise=true , address=145.79.227.49:4357 , piece_id=KXH4ZP22GRTNM2PUG56YHTQDSSOZTNLF4HZJ3ASPWUVSGK3U4SMA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760363528 (565h37m41.631446573s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760702464 (565h37m41.631785509s) [noise=true , address=145.79.227.12:4789 , piece_id=SLLU5XYDUV3TY7ETYJWXJHEZKA4RMLM5YVVEZG7YVPIVFGMYLL7Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.761056638 (565h37m41.632139673s) [noise=true , address=145.79.227.29:4975 , piece_id=TCVWZ65DXRLO4XXS4QURGXKGRFYXHQQSQY634FVU6TSGMTVC6IZQ, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.767898968 (565h37m41.638982013s) [noise=true , address=77.90.31.10:4525 , piece_id=CNBB6FZFANMK46XOWGIBIUNKF725QRRSIZPWIM2JRRX4RGMSHYIA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.769576983 (565h37m41.640660018s) [noise=true , address=145.79.227.14:4852 , piece_id=WDSJH7ALDJJLPTAANMI2E7JSLD7PBYKTJ32JQY7ILVORZUS6X7JA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.771126823 (565h37m41.642209888s) [noise=true , address=64.49.13.140:4514 , piece_id=VB4NOBKANE3OMUUURFJLSW6XZKVX32X4JTZRVKKTI26V7ZIDGR3Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.772734214 (565h37m41.643817259s) [noise=true , address=145.79.227.30:4796 , piece_id=OQGVOFUF6DY3Q3ZAOSOKLFLH47L4YMSW2SGOIFXYQ3TCXZLTJE6Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.774688165 (565h37m41.64577121s) [noise=true , address=77.90.31.13:4653 , piece_id=NGNJ2K55GTD3OFTYLDCUIT33OC5ZZ6QJYPF53HN62XXXKNGE64LA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.776343948 (565h37m41.647426983s) [noise=true , address=77.90.31.18:4830 , piece_id=MEJY5LQTTJV3R3MJSALXWU4THTD3XNZJJZ3Y7TQQOTIYMZBODMUA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.777891275 (565h37m41.64897431s) [noise=true , address=145.79.227.40:4072 , piece_id=SSWX7TAWXQKWBTFPBQR7FYQZFZVANVVVCVQJ45J46DS6XMGGEWCA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.779496952 (565h37m41.650579997s) [noise=true , address=145.79.227.4:4209 , piece_id=GYRCBERNVCY6BWYRRXK2GVHRCLFVXZIHD4JWBEAHNK3MKHDFDETA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:42.700027515 (565h37m42.57111055s) [noise=true , address=64.49.13.27:4959 , piece_id=6IHDQ6H64TI5Q653TPSGUTEZQYNABZCZO7AU26R2OLARLCUEPNZQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: ecclient: upload failed (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy, address:64.49.13.27:4959): protocol: storage node overloaded, request limit: 2000 Dec 29 05:06:43.064378199 (565h37m42.935461234s) [upload=495917]: Next returned nil segment Dec 29 05:06:43.064393940 (565h37m42.935476975s) [upload=495917]: Waiting for error group managing segments... Dec 29 05:06:43.492698353 (565h37m43.363781388s) [noise=true , address=64.49.13.140:4514 , piece_id=VB4NOBKANE3OMUUURFJLSW6XZKVX32X4JTZRVKKTI26V7ZIDGR3Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.620875843 (565h37m43.491958879s) [noise=true , address=145.79.227.12:4789 , piece_id=SLLU5XYDUV3TY7ETYJWXJHEZKA4RMLM5YVVEZG7YVPIVFGMYLL7Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.623709580 (565h37m43.494792635s) [noise=true , address=145.79.227.14:4852 , piece_id=WDSJH7ALDJJLPTAANMI2E7JSLD7PBYKTJ32JQY7ILVORZUS6X7JA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.634947790 (565h37m43.506030835s) [noise=true , address=145.79.227.4:4209 , piece_id=GYRCBERNVCY6BWYRRXK2GVHRCLFVXZIHD4JWBEAHNK3MKHDFDETA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.660936426 (565h37m43.532019461s) [noise=true , address=145.79.227.29:4975 , piece_id=TCVWZ65DXRLO4XXS4QURGXKGRFYXHQQSQY634FVU6TSGMTVC6IZQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.860684970 (565h37m43.731768016s) [noise=true , address=77.90.31.10:4525 , piece_id=CNBB6FZFANMK46XOWGIBIUNKF725QRRSIZPWIM2JRRX4RGMSHYIA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.897098368 (565h37m43.768181413s) [noise=true , address=77.90.31.13:4653 , piece_id=NGNJ2K55GTD3OFTYLDCUIT33OC5ZZ6QJYPF53HN62XXXKNGE64LA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.928905363 (565h37m43.799988408s) [noise=true , address=77.90.31.18:4830 , piece_id=MEJY5LQTTJV3R3MJSALXWU4THTD3XNZJJZ3Y7TQQOTIYMZBODMUA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.971303356 (565h37m43.842386401s) [noise=true , address=45.91.116.11:4124 , piece_id=FLAGSPMHPDJPGMBLHYBPDJKBWVKEQVEEZWVWZVAQQ6UHR6UYPEBA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.541477983 (565h37m44.412561018s) [noise=true , address=145.79.227.49:4357 , piece_id=KXH4ZP22GRTNM2PUG56YHTQDSSOZTNLF4HZJ3ASPWUVSGK3U4SMA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.554957591 (565h37m44.426040626s) [noise=true , address=145.79.227.43:4191 , piece_id=4GRQACHDAIEOXTFLCAQXS2ORNXMTOU6VSTE7YBAITEXRUWAO5JYQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.570831037 (565h37m44.441914072s) [noise=true , address=145.79.227.44:4220 , piece_id=5UMM6PIETJ2M5QLGNZNPQ6BZCC4QWG4HUX4YTFHX2ATAQJICH53A, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.580290328 (565h37m44.451373363s) [noise=true , address=145.79.227.40:4072 , piece_id=SSWX7TAWXQKWBTFPBQR7FYQZFZVANVVVCVQJ45J46DS6XMGGEWCA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.655455876 (565h37m44.526538911s) [noise=true , address=145.79.227.30:4796 , piece_id=OQGVOFUF6DY3Q3ZAOSOKLFLH47L4YMSW2SGOIFXYQ3TCXZLTJE6Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.655480864 (565h37m44.526563909s) [seg_pos={0 0}, upload=495917]: Segment reached optimal threshold of 14 pieces. Dec 29 05:06:44.655678942 (565h37m44.526761977s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:44.655686626 (565h37m44.526769671s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.655850348 (565h37m44.526933383s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection Dec 29 05:06:44.655857702 (565h37m44.526940747s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.656053324 (565h37m44.527136369s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:44.656061891 (565h37m44.527144926s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.656092720 (565h37m44.527175765s) [seg_pos={0 0}, upload=495917]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:06:44.656214661 (565h37m44.527297696s) [upload=495917]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:06:44.656221103 (565h37m44.527304149s) [upload=495917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:06:44.975569917 (565h37m44.846652962s) [upload=495917]: Done uploading segments... Dec 29 05:06:44.975596969 (565h37m44.846680004s) [upload=495917]: Upload finished. err: <nil> Duration: 34.730376613s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.....XX.......X.XX..XX..XXX........XX......X...X..XXX..X.] duration: 6.248302972s trace: https://jaeger.dev.storj.io/trace/166375b545182e5f FAIL (2025-12-29 05:07:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q) context deadline exceeded ======LOGS====== 29 05:06:36.970550676 (565h37m36.841633711s) [upload=495915]: Starting upload Dec 29 05:06:36.970689340 (565h37m36.841772385s) [upload=495915]: Done starting upload Dec 29 05:06:36.970766808 (565h37m36.841849843s) [upload=495915]: Uploading segments... Dec 29 05:06:36.970956749 (565h37m36.842039784s) [upload=495915]: Got next segment. Inline: false Dec 29 05:06:36.971001284 (565h37m36.842084319s) [upload=495915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:06:36.971007365 (565h37m36.8420904s) [upload=495915]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:06:37.219391530 (565h37m37.090474565s) [seg_pos={0 0}, upload=495915]: Begin upload segment... Dec 29 05:06:37.219442347 (565h37m37.090525382s) [seg_pos={0 0}, upload=495915]: Done begin upload segment. Dec 29 05:06:37.219473056 (565h37m37.090556091s) [noise=true , address=77.90.31.12:4589 , piece_id=LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.219523822 (565h37m37.090606867s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220113205 (565h37m37.09119625s) [noise=true , address=77.90.31.14:4663 , piece_id=ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220540107 (565h37m37.091623142s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220551049 (565h37m37.091634094s) [noise=true , address=145.79.227.29:4942 , piece_id=ZG3PNUCUXGUA2QP4LEWPQS7GOYYPAHXAKXMZDNKOJCP2BWS5KHKQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220947343 (565h37m37.092030378s) [noise=true , address=145.79.227.50:4374 , piece_id=HSIGXR74IPP6SOW7LGVWUWPOG3ALODIE5MM7BYNCX7VZGUHLJHCA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221017587 (565h37m37.092100632s) [noise=true , address=145.79.227.13:4804 , piece_id=SX6GJDLYSEIO2VMC3CZ3QXE5ED74AOYQFWB6ILJ6HBXTVODUBQZA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221232225 (565h37m37.09231526s) [noise=true , address=77.90.31.11:4555 , piece_id=O4PLT2VBZD3MHESP6BNYD4P6KPB3TWQO7INKBQ3K7VXUVSU2OYXA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221437186 (565h37m37.092520231s) [noise=true , address=145.79.227.46:4602 , piece_id=CMVURWNPAEJFFW7OLNI3LGT5ERT2CVHBAJ32WRF5NRC5KUEWPUJA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221512349 (565h37m37.092595394s) [noise=true , address=45.91.116.4:4937 , piece_id=47MMITQEIU2ROUVT3DGLXUKW2KQTYQKTL3B2JXTKISBGAJVGFGYQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221787312 (565h37m37.092870347s) [noise=true , address=45.91.116.13:4156 , piece_id=NFXHBJZRTUTLPCCOSIVHPFQEL54ZLRW7VRYGJ3RGEPUHMQYAF73Q, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222126538 (565h37m37.093209574s) [noise=true , address=145.79.227.42:4123 , piece_id=C3J47FCY5NHBU2EVOYJWL2TXIZM3T2NP4QJR7DCXA5OE66I2MTJA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222409818 (565h37m37.093492863s) [noise=true , address=145.79.227.47:4266 , piece_id=5EIC6KID2TXVUISCFF2JGVP2ROEQWBFDVWIZMJ4PLYO2E4ZQ4VFA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222766296 (565h37m37.093849331s) [noise=true , address=145.79.227.24:4582 , piece_id=BYRUYICBLZPAWQMT3VH6OPDOW2M7T7RT2BQ7KCYVML2T3X77UGQQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.223062421 (565h37m37.094145466s) [noise=true , address=145.79.227.52:4446 , piece_id=G2AOIQ6NGVCGZ5HVFWUBJC7FCSDGRZ3QUVWT5Y2XCLF3BLC52EFQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.223388491 (565h37m37.094471526s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:38.912022873 (565h37m38.783105908s) [upload=495915]: Next returned nil segment Dec 29 05:06:38.912038503 (565h37m38.783121538s) [upload=495915]: Waiting for error group managing segments... Dec 29 05:06:39.196335632 (565h37m39.067418667s) [noise=true , address=145.79.227.13:4804 , piece_id=SX6GJDLYSEIO2VMC3CZ3QXE5ED74AOYQFWB6ILJ6HBXTVODUBQZA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.211162354 (565h37m39.08224539s) [noise=true , address=145.79.227.52:4446 , piece_id=G2AOIQ6NGVCGZ5HVFWUBJC7FCSDGRZ3QUVWT5Y2XCLF3BLC52EFQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.212495473 (565h37m39.083578508s) [noise=true , address=145.79.227.42:4123 , piece_id=C3J47FCY5NHBU2EVOYJWL2TXIZM3T2NP4QJR7DCXA5OE66I2MTJA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.213852737 (565h37m39.084935772s) [noise=true , address=145.79.227.50:4374 , piece_id=HSIGXR74IPP6SOW7LGVWUWPOG3ALODIE5MM7BYNCX7VZGUHLJHCA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.218895510 (565h37m39.089978545s) [noise=true , address=145.79.227.47:4266 , piece_id=5EIC6KID2TXVUISCFF2JGVP2ROEQWBFDVWIZMJ4PLYO2E4ZQ4VFA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.226304798 (565h37m39.097387843s) [noise=true , address=145.79.227.24:4582 , piece_id=BYRUYICBLZPAWQMT3VH6OPDOW2M7T7RT2BQ7KCYVML2T3X77UGQQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.227391057 (565h37m39.098474102s) [noise=true , address=145.79.227.46:4602 , piece_id=CMVURWNPAEJFFW7OLNI3LGT5ERT2CVHBAJ32WRF5NRC5KUEWPUJA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.360080590 (565h37m39.231163625s) [noise=true , address=145.79.227.29:4942 , piece_id=ZG3PNUCUXGUA2QP4LEWPQS7GOYYPAHXAKXMZDNKOJCP2BWS5KHKQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.402523819 (565h37m39.273606854s) [noise=true , address=77.90.31.14:4663 , piece_id=ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.423611764 (565h37m39.294694809s) [noise=true , address=77.90.31.11:4555 , piece_id=O4PLT2VBZD3MHESP6BNYD4P6KPB3TWQO7INKBQ3K7VXUVSU2OYXA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.492584129 (565h37m39.363667164s) [noise=true , address=45.91.116.4:4937 , piece_id=47MMITQEIU2ROUVT3DGLXUKW2KQTYQKTL3B2JXTKISBGAJVGFGYQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.500637974 (565h37m39.371721009s) [noise=true , address=45.91.116.13:4156 , piece_id=NFXHBJZRTUTLPCCOSIVHPFQEL54ZLRW7VRYGJ3RGEPUHMQYAF73Q, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:40.183710850 (565h37m40.054793885s) [noise=true , address=77.90.31.12:4589 , piece_id=LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:40.183730306 (565h37m40.054813351s) [seg_pos={0 0}, upload=495915]: Segment reached optimal threshold of 13 pieces. Dec 29 05:06:40.183885813 (565h37m40.054968848s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:40.183892525 (565h37m40.05497557s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184152319 (565h37m40.055235355s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection Dec 29 05:06:40.184158902 (565h37m40.055241947s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184268852 (565h37m40.055351897s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:40.184274713 (565h37m40.055357748s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184290533 (565h37m40.055373578s) [seg_pos={0 0}, upload=495915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:06:40.184360165 (565h37m40.05544321s) [upload=495915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:06:40.184365886 (565h37m40.055448931s) [upload=495915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:06:40.481038220 (565h37m40.352121275s) [upload=495915]: Done uploading segments... Dec 29 05:06:40.481063508 (565h37m40.352146553s) [upload=495915]: Upload finished. err: <nil> Duration: 6.248334069s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....X.....X..XX......X.X.....X.....X........X..X...X....X..] duration: 4.717155841s trace: https://jaeger.dev.storj.io/trace/2cf94efd24994e89 FAIL (2025-12-29 05:07:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.717186801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......X.....X..XX......X.X.....X.....X........X..X...X....X.] duration: 3.527055461s trace: https://jaeger.dev.storj.io/trace/2cf94efd24994e89 FAIL (2025-12-29 05:07:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527086059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 33.426114011s trace: https://jaeger.dev.storj.io/trace/1935d16c4ec3317c FAIL (2025-12-29 05:07:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 6:41.757908043 (565h37m41.628991078s) [noise=true , address=64.49.13.27:4959 , piece_id=6IHDQ6H64TI5Q653TPSGUTEZQYNABZCZO7AU26R2OLARLCUEPNZQ, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.758607134 (565h37m41.629690189s) [noise=true , address=145.79.227.44:4220 , piece_id=5UMM6PIETJ2M5QLGNZNPQ6BZCC4QWG4HUX4YTFHX2ATAQJICH53A, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.758997888 (565h37m41.630080933s) [noise=true , address=45.91.116.11:4124 , piece_id=FLAGSPMHPDJPGMBLHYBPDJKBWVKEQVEEZWVWZVAQQ6UHR6UYPEBA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.759288822 (565h37m41.630371867s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.759638187 (565h37m41.630721222s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760079808 (565h37m41.631162853s) [noise=true , address=145.79.227.49:4357 , piece_id=KXH4ZP22GRTNM2PUG56YHTQDSSOZTNLF4HZJ3ASPWUVSGK3U4SMA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760363528 (565h37m41.631446573s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.760702464 (565h37m41.631785509s) [noise=true , address=145.79.227.12:4789 , piece_id=SLLU5XYDUV3TY7ETYJWXJHEZKA4RMLM5YVVEZG7YVPIVFGMYLL7Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.761056638 (565h37m41.632139673s) [noise=true , address=145.79.227.29:4975 , piece_id=TCVWZ65DXRLO4XXS4QURGXKGRFYXHQQSQY634FVU6TSGMTVC6IZQ, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.767898968 (565h37m41.638982013s) [noise=true , address=77.90.31.10:4525 , piece_id=CNBB6FZFANMK46XOWGIBIUNKF725QRRSIZPWIM2JRRX4RGMSHYIA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.769576983 (565h37m41.640660018s) [noise=true , address=145.79.227.14:4852 , piece_id=WDSJH7ALDJJLPTAANMI2E7JSLD7PBYKTJ32JQY7ILVORZUS6X7JA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.771126823 (565h37m41.642209888s) [noise=true , address=64.49.13.140:4514 , piece_id=VB4NOBKANE3OMUUURFJLSW6XZKVX32X4JTZRVKKTI26V7ZIDGR3Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.772734214 (565h37m41.643817259s) [noise=true , address=145.79.227.30:4796 , piece_id=OQGVOFUF6DY3Q3ZAOSOKLFLH47L4YMSW2SGOIFXYQ3TCXZLTJE6Q, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.774688165 (565h37m41.64577121s) [noise=true , address=77.90.31.13:4653 , piece_id=NGNJ2K55GTD3OFTYLDCUIT33OC5ZZ6QJYPF53HN62XXXKNGE64LA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.776343948 (565h37m41.647426983s) [noise=true , address=77.90.31.18:4830 , piece_id=MEJY5LQTTJV3R3MJSALXWU4THTD3XNZJJZ3Y7TQQOTIYMZBODMUA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.777891275 (565h37m41.64897431s) [noise=true , address=145.79.227.40:4072 , piece_id=SSWX7TAWXQKWBTFPBQR7FYQZFZVANVVVCVQJ45J46DS6XMGGEWCA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:41.779496952 (565h37m41.650579997s) [noise=true , address=145.79.227.4:4209 , piece_id=GYRCBERNVCY6BWYRRXK2GVHRCLFVXZIHD4JWBEAHNK3MKHDFDETA, seg_pos={0 0}, upload=495917]: Uploading piece... Dec 29 05:06:42.700027515 (565h37m42.57111055s) [noise=true , address=64.49.13.27:4959 , piece_id=6IHDQ6H64TI5Q653TPSGUTEZQYNABZCZO7AU26R2OLARLCUEPNZQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: ecclient: upload failed (node:1XwcXkyKfaFNtGQpfU6npzHYAz2ySySmvdT9ya9HUa7u6dR2Zy, address:64.49.13.27:4959): protocol: storage node overloaded, request limit: 2000 Dec 29 05:06:43.064378199 (565h37m42.935461234s) [upload=495917]: Next returned nil segment Dec 29 05:06:43.064393940 (565h37m42.935476975s) [upload=495917]: Waiting for error group managing segments... Dec 29 05:06:43.492698353 (565h37m43.363781388s) [noise=true , address=64.49.13.140:4514 , piece_id=VB4NOBKANE3OMUUURFJLSW6XZKVX32X4JTZRVKKTI26V7ZIDGR3Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.620875843 (565h37m43.491958879s) [noise=true , address=145.79.227.12:4789 , piece_id=SLLU5XYDUV3TY7ETYJWXJHEZKA4RMLM5YVVEZG7YVPIVFGMYLL7Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.623709580 (565h37m43.494792635s) [noise=true , address=145.79.227.14:4852 , piece_id=WDSJH7ALDJJLPTAANMI2E7JSLD7PBYKTJ32JQY7ILVORZUS6X7JA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.634947790 (565h37m43.506030835s) [noise=true , address=145.79.227.4:4209 , piece_id=GYRCBERNVCY6BWYRRXK2GVHRCLFVXZIHD4JWBEAHNK3MKHDFDETA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.660936426 (565h37m43.532019461s) [noise=true , address=145.79.227.29:4975 , piece_id=TCVWZ65DXRLO4XXS4QURGXKGRFYXHQQSQY634FVU6TSGMTVC6IZQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.860684970 (565h37m43.731768016s) [noise=true , address=77.90.31.10:4525 , piece_id=CNBB6FZFANMK46XOWGIBIUNKF725QRRSIZPWIM2JRRX4RGMSHYIA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.897098368 (565h37m43.768181413s) [noise=true , address=77.90.31.13:4653 , piece_id=NGNJ2K55GTD3OFTYLDCUIT33OC5ZZ6QJYPF53HN62XXXKNGE64LA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.928905363 (565h37m43.799988408s) [noise=true , address=77.90.31.18:4830 , piece_id=MEJY5LQTTJV3R3MJSALXWU4THTD3XNZJJZ3Y7TQQOTIYMZBODMUA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:43.971303356 (565h37m43.842386401s) [noise=true , address=45.91.116.11:4124 , piece_id=FLAGSPMHPDJPGMBLHYBPDJKBWVKEQVEEZWVWZVAQQ6UHR6UYPEBA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.541477983 (565h37m44.412561018s) [noise=true , address=145.79.227.49:4357 , piece_id=KXH4ZP22GRTNM2PUG56YHTQDSSOZTNLF4HZJ3ASPWUVSGK3U4SMA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.554957591 (565h37m44.426040626s) [noise=true , address=145.79.227.43:4191 , piece_id=4GRQACHDAIEOXTFLCAQXS2ORNXMTOU6VSTE7YBAITEXRUWAO5JYQ, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.570831037 (565h37m44.441914072s) [noise=true , address=145.79.227.44:4220 , piece_id=5UMM6PIETJ2M5QLGNZNPQ6BZCC4QWG4HUX4YTFHX2ATAQJICH53A, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.580290328 (565h37m44.451373363s) [noise=true , address=145.79.227.40:4072 , piece_id=SSWX7TAWXQKWBTFPBQR7FYQZFZVANVVVCVQJ45J46DS6XMGGEWCA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.655455876 (565h37m44.526538911s) [noise=true , address=145.79.227.30:4796 , piece_id=OQGVOFUF6DY3Q3ZAOSOKLFLH47L4YMSW2SGOIFXYQ3TCXZLTJE6Q, seg_pos={0 0}, upload=495917]: Done uploading piece. err: <nil> Dec 29 05:06:44.655480864 (565h37m44.526563909s) [seg_pos={0 0}, upload=495917]: Segment reached optimal threshold of 14 pieces. Dec 29 05:06:44.655678942 (565h37m44.526761977s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tViqrJYdX8dCfHMjVU9sDatzUARxbGqvvt1x1C334N5SXoaRu): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:44.655686626 (565h37m44.526769671s) [noise=true , address=145.79.227.34:4919 , piece_id=DLRJJ7OZT5SK6PEN6FO75ME4CHUGR45FHBZAFBJQQC6MTNZY7BKA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.655850348 (565h37m44.526933383s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59700->64.49.13.169:4711: use of closed network connection Dec 29 05:06:44.655857702 (565h37m44.526940747s) [noise=true , address=64.49.13.169:4711 , piece_id=Z7BIRZZUXEDR7ZFLEVQDMBOWTXURVKRUSX7D5I2NZTDNQZLQGMIA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.656053324 (565h37m44.527136369s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:44.656061891 (565h37m44.527144926s) [noise=true , address=145.79.227.35:4506 , piece_id=EIFHES6DOROTVS26XPJNV3KSJE2E2PI4INSDOI7JZQHPXAMKRZPA, seg_pos={0 0}, upload=495917]: Long tail cancellation due to optimal threshold Dec 29 05:06:44.656092720 (565h37m44.527175765s) [seg_pos={0 0}, upload=495917]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 05:06:44.656214661 (565h37m44.527297696s) [upload=495917]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:06:44.656221103 (565h37m44.527304149s) [upload=495917]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:06:44.975569917 (565h37m44.846652962s) [upload=495917]: Done uploading segments... Dec 29 05:06:44.975596969 (565h37m44.846680004s) [upload=495917]: Upload finished. err: <nil> Duration: 33.426167993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.X.....XX.......X.XX..XX..XXX........XX......X...X..XXX..X] duration: 30.00210163s trace: https://jaeger.dev.storj.io/trace/166375b545182e5f FAIL (2025-12-29 05:07:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q) context deadline exceeded ======LOGS====== 29 05:06:36.970550676 (565h37m36.841633711s) [upload=495915]: Starting upload Dec 29 05:06:36.970689340 (565h37m36.841772385s) [upload=495915]: Done starting upload Dec 29 05:06:36.970766808 (565h37m36.841849843s) [upload=495915]: Uploading segments... Dec 29 05:06:36.970956749 (565h37m36.842039784s) [upload=495915]: Got next segment. Inline: false Dec 29 05:06:36.971001284 (565h37m36.842084319s) [upload=495915]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:06:36.971007365 (565h37m36.8420904s) [upload=495915]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:06:37.219391530 (565h37m37.090474565s) [seg_pos={0 0}, upload=495915]: Begin upload segment... Dec 29 05:06:37.219442347 (565h37m37.090525382s) [seg_pos={0 0}, upload=495915]: Done begin upload segment. Dec 29 05:06:37.219473056 (565h37m37.090556091s) [noise=true , address=77.90.31.12:4589 , piece_id=LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.219523822 (565h37m37.090606867s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220113205 (565h37m37.09119625s) [noise=true , address=77.90.31.14:4663 , piece_id=ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220540107 (565h37m37.091623142s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220551049 (565h37m37.091634094s) [noise=true , address=145.79.227.29:4942 , piece_id=ZG3PNUCUXGUA2QP4LEWPQS7GOYYPAHXAKXMZDNKOJCP2BWS5KHKQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.220947343 (565h37m37.092030378s) [noise=true , address=145.79.227.50:4374 , piece_id=HSIGXR74IPP6SOW7LGVWUWPOG3ALODIE5MM7BYNCX7VZGUHLJHCA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221017587 (565h37m37.092100632s) [noise=true , address=145.79.227.13:4804 , piece_id=SX6GJDLYSEIO2VMC3CZ3QXE5ED74AOYQFWB6ILJ6HBXTVODUBQZA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221232225 (565h37m37.09231526s) [noise=true , address=77.90.31.11:4555 , piece_id=O4PLT2VBZD3MHESP6BNYD4P6KPB3TWQO7INKBQ3K7VXUVSU2OYXA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221437186 (565h37m37.092520231s) [noise=true , address=145.79.227.46:4602 , piece_id=CMVURWNPAEJFFW7OLNI3LGT5ERT2CVHBAJ32WRF5NRC5KUEWPUJA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221512349 (565h37m37.092595394s) [noise=true , address=45.91.116.4:4937 , piece_id=47MMITQEIU2ROUVT3DGLXUKW2KQTYQKTL3B2JXTKISBGAJVGFGYQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.221787312 (565h37m37.092870347s) [noise=true , address=45.91.116.13:4156 , piece_id=NFXHBJZRTUTLPCCOSIVHPFQEL54ZLRW7VRYGJ3RGEPUHMQYAF73Q, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222126538 (565h37m37.093209574s) [noise=true , address=145.79.227.42:4123 , piece_id=C3J47FCY5NHBU2EVOYJWL2TXIZM3T2NP4QJR7DCXA5OE66I2MTJA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222409818 (565h37m37.093492863s) [noise=true , address=145.79.227.47:4266 , piece_id=5EIC6KID2TXVUISCFF2JGVP2ROEQWBFDVWIZMJ4PLYO2E4ZQ4VFA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.222766296 (565h37m37.093849331s) [noise=true , address=145.79.227.24:4582 , piece_id=BYRUYICBLZPAWQMT3VH6OPDOW2M7T7RT2BQ7KCYVML2T3X77UGQQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.223062421 (565h37m37.094145466s) [noise=true , address=145.79.227.52:4446 , piece_id=G2AOIQ6NGVCGZ5HVFWUBJC7FCSDGRZ3QUVWT5Y2XCLF3BLC52EFQ, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:37.223388491 (565h37m37.094471526s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Uploading piece... Dec 29 05:06:38.912022873 (565h37m38.783105908s) [upload=495915]: Next returned nil segment Dec 29 05:06:38.912038503 (565h37m38.783121538s) [upload=495915]: Waiting for error group managing segments... Dec 29 05:06:39.196335632 (565h37m39.067418667s) [noise=true , address=145.79.227.13:4804 , piece_id=SX6GJDLYSEIO2VMC3CZ3QXE5ED74AOYQFWB6ILJ6HBXTVODUBQZA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.211162354 (565h37m39.08224539s) [noise=true , address=145.79.227.52:4446 , piece_id=G2AOIQ6NGVCGZ5HVFWUBJC7FCSDGRZ3QUVWT5Y2XCLF3BLC52EFQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.212495473 (565h37m39.083578508s) [noise=true , address=145.79.227.42:4123 , piece_id=C3J47FCY5NHBU2EVOYJWL2TXIZM3T2NP4QJR7DCXA5OE66I2MTJA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.213852737 (565h37m39.084935772s) [noise=true , address=145.79.227.50:4374 , piece_id=HSIGXR74IPP6SOW7LGVWUWPOG3ALODIE5MM7BYNCX7VZGUHLJHCA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.218895510 (565h37m39.089978545s) [noise=true , address=145.79.227.47:4266 , piece_id=5EIC6KID2TXVUISCFF2JGVP2ROEQWBFDVWIZMJ4PLYO2E4ZQ4VFA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.226304798 (565h37m39.097387843s) [noise=true , address=145.79.227.24:4582 , piece_id=BYRUYICBLZPAWQMT3VH6OPDOW2M7T7RT2BQ7KCYVML2T3X77UGQQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.227391057 (565h37m39.098474102s) [noise=true , address=145.79.227.46:4602 , piece_id=CMVURWNPAEJFFW7OLNI3LGT5ERT2CVHBAJ32WRF5NRC5KUEWPUJA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.360080590 (565h37m39.231163625s) [noise=true , address=145.79.227.29:4942 , piece_id=ZG3PNUCUXGUA2QP4LEWPQS7GOYYPAHXAKXMZDNKOJCP2BWS5KHKQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.402523819 (565h37m39.273606854s) [noise=true , address=77.90.31.14:4663 , piece_id=ICXUUNPWWBGQEOMARVENQQWNQHEH4SNOPU4CY6CGO4Z2RN3QC57Q, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.423611764 (565h37m39.294694809s) [noise=true , address=77.90.31.11:4555 , piece_id=O4PLT2VBZD3MHESP6BNYD4P6KPB3TWQO7INKBQ3K7VXUVSU2OYXA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.492584129 (565h37m39.363667164s) [noise=true , address=45.91.116.4:4937 , piece_id=47MMITQEIU2ROUVT3DGLXUKW2KQTYQKTL3B2JXTKISBGAJVGFGYQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:39.500637974 (565h37m39.371721009s) [noise=true , address=45.91.116.13:4156 , piece_id=NFXHBJZRTUTLPCCOSIVHPFQEL54ZLRW7VRYGJ3RGEPUHMQYAF73Q, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:40.183710850 (565h37m40.054793885s) [noise=true , address=77.90.31.12:4589 , piece_id=LAMTE5ENJST5RIDTCMOVWBYVXOK3VIGPB5FK3WVG4TADCGA2UBZQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: <nil> Dec 29 05:06:40.183730306 (565h37m40.054813351s) [seg_pos={0 0}, upload=495915]: Segment reached optimal threshold of 13 pieces. Dec 29 05:06:40.183885813 (565h37m40.054968848s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:40.183892525 (565h37m40.05497557s) [noise=true , address=145.79.227.49:4338 , piece_id=QDITTZOITNKTOGNVU6YD5WELHRZLGDPTGW6ILAWZKMTOPSY4N2IA, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184152319 (565h37m40.055235355s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51420->45.91.116.12:4228: use of closed network connection Dec 29 05:06:40.184158902 (565h37m40.055241947s) [noise=true , address=45.91.116.12:4228 , piece_id=OCMRAZ2BZ6HXOKOERGBJO4U2OWNJNC6XX6ZXAZ6EH3C7TSACQCWQ, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184268852 (565h37m40.055351897s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 29 05:06:40.184274713 (565h37m40.055357748s) [noise=true , address=45.91.116.8:4048 , piece_id=RTU3F5N3FS475ED4IXWOSJIUAHKMTT6THXC4E5TH2M6IGSZFESAQ, seg_pos={0 0}, upload=495915]: Long tail cancellation due to optimal threshold Dec 29 05:06:40.184290533 (565h37m40.055373578s) [seg_pos={0 0}, upload=495915]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:06:40.184360165 (565h37m40.05544321s) [upload=495915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:06:40.184365886 (565h37m40.055448931s) [upload=495915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:06:40.481038220 (565h37m40.352121275s) [upload=495915]: Done uploading segments... Dec 29 05:06:40.481063508 (565h37m40.352146553s) [upload=495915]: Upload finished. err: <nil> Duration: 30.002115986s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......X.....X..XX......X.X.....X.....X........X..X...X....X] duration: 5.0017995s trace: https://jaeger.dev.storj.io/trace/2cf94efd24994e89 FAIL (2025-12-29 05:06:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001820008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.X.....XX.......X.XX..XX..XXX........XX......X...X..XXX..] duration: 5.935813919s trace: https://jaeger.dev.storj.io/trace/17ba19948c5ed686 FAIL (2025-12-29 05:06:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ) context deadline exceeded ======LOGS====== Dec 29 05:05:18.181732997 (565h36m18.052816032s) [upload=495898]: Starting upload Dec 29 05:05:18.181790907 (565h36m18.052873942s) [upload=495898]: Done starting upload Dec 29 05:05:18.181950892 (565h36m18.053033937s) [upload=495898]: Uploading segments... Dec 29 05:05:18.182057124 (565h36m18.053140169s) [upload=495898]: Got next segment. Inline: false Dec 29 05:05:18.182102791 (565h36m18.053185826s) [upload=495898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:05:18.182109564 (565h36m18.053192609s) [upload=495898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:05:18.447800274 (565h36m18.318883329s) [seg_pos={0 0}, upload=495898]: Begin upload segment... Dec 29 05:05:18.447877992 (565h36m18.318961037s) [seg_pos={0 0}, upload=495898]: Done begin upload segment. Dec 29 05:05:18.447917717 (565h36m18.319000762s) [noise=true , address=145.79.227.39:4015 , piece_id=BLVDCPOGAE2MUQQMB6XDMA73KIDTZ7HYIKZJZ733BYEV3YFIDYHQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.448199444 (565h36m18.31928249s) [noise=true , address=145.79.227.11:4729 , piece_id=4QF6AFGC4O7KQIBEJBPX3SQFZGVXOSLAQBFIRY5X6UGYIXC6JCVQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.448758288 (565h36m18.319841333s) [noise=true , address=145.79.227.46:4602 , piece_id=2WRL4WV3TUZWJ4QBCHPX3XS5A4HXMM6FMPDNR5KMQ7P3KC3CRK7A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449100429 (565h36m18.320183464s) [noise=true , address=145.79.227.7:4295 , piece_id=RD56LWBXLXY7H7UC4QRRBAEIZ6GBO2LLUPAJDK7W42HQB77P2SDA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449440938 (565h36m18.320523983s) [noise=true , address=77.90.31.14:4661 , piece_id=CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449781406 (565h36m18.320864451s) [noise=true , address=45.91.116.12:4227 , piece_id=4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450112496 (565h36m18.321195541s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450191407 (565h36m18.321274452s) [noise=true , address=145.79.227.26:4655 , piece_id=AD6TNQDFKZCXJRBEDOTBHRILASW2JM6QJHLA56ULAKWQPCDFGC3A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450401998 (565h36m18.321485043s) [noise=true , address=45.91.116.5:4297 , piece_id=QUYLCQ7XKDFKUQU4KI3J5OB3ECMWBJ4TZ3RY7P2S4EE6K2XBW4SQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450974518 (565h36m18.322057814s) [noise=true , address=145.79.227.47:4267 , piece_id=XMN77OZAPUEB4NUYGJHAMTOW6GLEOE7G3BOAMWI3ZUDVNUWJYTTQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.451548161 (565h36m18.322631256s) [noise=true , address=77.90.31.17:4770 , piece_id=NPBLI7RMBP5BAOAYEBGEIH7YWRMFEOSNVPAVLSRTJSR6UQ66F4KA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452199420 (565h36m18.323282475s) [noise=true , address=145.79.227.15:4876 , piece_id=IMII2KVEAQMMLZUC5KE42YTRE37PI2JP7K4MMVUMNJVMEK4OS76Q, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452279543 (565h36m18.323362579s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452667662 (565h36m18.323750707s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.453060510 (565h36m18.324143555s) [noise=true , address=145.79.227.18:4007 , piece_id=M7RXQXZI4UJUQ6CMOEVFG3FHWOEGF6P4LN3NCVPOHWPO7Z65EWMA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.453574219 (565h36m18.324657264s) [noise=true , address=145.79.227.4:4185 , piece_id=X2LRCXCTDCSOXPJNZDD5SDLXRBU5MU6GLBX3DODHIVP3XFW5QBKQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:20.138758440 (565h36m20.009841475s) [upload=495898]: Next returned nil segment Dec 29 05:05:20.138775443 (565h36m20.009858488s) [upload=495898]: Waiting for error group managing segments... Dec 29 05:05:20.417061276 (565h36m20.288144321s) [noise=true , address=145.79.227.7:4295 , piece_id=RD56LWBXLXY7H7UC4QRRBAEIZ6GBO2LLUPAJDK7W42HQB77P2SDA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.438373497 (565h36m20.309456542s) [noise=true , address=145.79.227.46:4602 , piece_id=2WRL4WV3TUZWJ4QBCHPX3XS5A4HXMM6FMPDNR5KMQ7P3KC3CRK7A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.438983670 (565h36m20.310066715s) [noise=true , address=145.79.227.15:4876 , piece_id=IMII2KVEAQMMLZUC5KE42YTRE37PI2JP7K4MMVUMNJVMEK4OS76Q, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.439283922 (565h36m20.310366957s) [noise=true , address=145.79.227.39:4015 , piece_id=BLVDCPOGAE2MUQQMB6XDMA73KIDTZ7HYIKZJZ733BYEV3YFIDYHQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.440250381 (565h36m20.311333426s) [noise=true , address=145.79.227.11:4729 , piece_id=4QF6AFGC4O7KQIBEJBPX3SQFZGVXOSLAQBFIRY5X6UGYIXC6JCVQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.454050149 (565h36m20.325133184s) [noise=true , address=145.79.227.4:4185 , piece_id=X2LRCXCTDCSOXPJNZDD5SDLXRBU5MU6GLBX3DODHIVP3XFW5QBKQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.455700752 (565h36m20.326783797s) [noise=true , address=145.79.227.47:4267 , piece_id=XMN77OZAPUEB4NUYGJHAMTOW6GLEOE7G3BOAMWI3ZUDVNUWJYTTQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.459085748 (565h36m20.330168783s) [noise=true , address=145.79.227.26:4655 , piece_id=AD6TNQDFKZCXJRBEDOTBHRILASW2JM6QJHLA56ULAKWQPCDFGC3A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.459918964 (565h36m20.331001999s) [noise=true , address=145.79.227.18:4007 , piece_id=M7RXQXZI4UJUQ6CMOEVFG3FHWOEGF6P4LN3NCVPOHWPO7Z65EWMA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.704694136 (565h36m20.575777171s) [noise=true , address=77.90.31.14:4661 , piece_id=CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.791101300 (565h36m20.662184345s) [noise=true , address=45.91.116.12:4227 , piece_id=4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.837150746 (565h36m20.708233791s) [noise=true , address=77.90.31.17:4770 , piece_id=NPBLI7RMBP5BAOAYEBGEIH7YWRMFEOSNVPAVLSRTJSR6UQ66F4KA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.843066351 (565h36m20.714149386s) [noise=true , address=45.91.116.5:4297 , piece_id=QUYLCQ7XKDFKUQU4KI3J5OB3ECMWBJ4TZ3RY7P2S4EE6K2XBW4SQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.843089795 (565h36m20.71417284s) [seg_pos={0 0}, upload=495898]: Segment reached optimal threshold of 13 pieces. Dec 29 05:05:20.843263455 (565h36m20.71434649s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843269638 (565h36m20.714352673s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843538850 (565h36m20.714621885s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843545442 (565h36m20.714628477s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843651455 (565h36m20.7147345s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843657116 (565h36m20.714740161s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843673486 (565h36m20.714756521s) [seg_pos={0 0}, upload=495898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:05:20.843744972 (565h36m20.714828018s) [upload=495898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:05:20.843749932 (565h36m20.714832967s) [upload=495898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:05:21.143609134 (565h36m21.01469218s) [upload=495898]: Done uploading segments... Dec 29 05:05:21.143634652 (565h36m21.014717697s) [upload=495898]: Upload finished. err: <nil> Duration: 5.935839238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..XX.X.....XX.......X.XX..XX..XXX........XX......X...X..XXX.] duration: 6.036808347s trace: https://jaeger.dev.storj.io/trace/17ba19948c5ed686 FAIL (2025-12-29 05:05:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ) context deadline exceeded ======LOGS====== Dec 29 05:05:18.181732997 (565h36m18.052816032s) [upload=495898]: Starting upload Dec 29 05:05:18.181790907 (565h36m18.052873942s) [upload=495898]: Done starting upload Dec 29 05:05:18.181950892 (565h36m18.053033937s) [upload=495898]: Uploading segments... Dec 29 05:05:18.182057124 (565h36m18.053140169s) [upload=495898]: Got next segment. Inline: false Dec 29 05:05:18.182102791 (565h36m18.053185826s) [upload=495898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:05:18.182109564 (565h36m18.053192609s) [upload=495898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:05:18.447800274 (565h36m18.318883329s) [seg_pos={0 0}, upload=495898]: Begin upload segment... Dec 29 05:05:18.447877992 (565h36m18.318961037s) [seg_pos={0 0}, upload=495898]: Done begin upload segment. Dec 29 05:05:18.447917717 (565h36m18.319000762s) [noise=true , address=145.79.227.39:4015 , piece_id=BLVDCPOGAE2MUQQMB6XDMA73KIDTZ7HYIKZJZ733BYEV3YFIDYHQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.448199444 (565h36m18.31928249s) [noise=true , address=145.79.227.11:4729 , piece_id=4QF6AFGC4O7KQIBEJBPX3SQFZGVXOSLAQBFIRY5X6UGYIXC6JCVQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.448758288 (565h36m18.319841333s) [noise=true , address=145.79.227.46:4602 , piece_id=2WRL4WV3TUZWJ4QBCHPX3XS5A4HXMM6FMPDNR5KMQ7P3KC3CRK7A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449100429 (565h36m18.320183464s) [noise=true , address=145.79.227.7:4295 , piece_id=RD56LWBXLXY7H7UC4QRRBAEIZ6GBO2LLUPAJDK7W42HQB77P2SDA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449440938 (565h36m18.320523983s) [noise=true , address=77.90.31.14:4661 , piece_id=CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449781406 (565h36m18.320864451s) [noise=true , address=45.91.116.12:4227 , piece_id=4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450112496 (565h36m18.321195541s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450191407 (565h36m18.321274452s) [noise=true , address=145.79.227.26:4655 , piece_id=AD6TNQDFKZCXJRBEDOTBHRILASW2JM6QJHLA56ULAKWQPCDFGC3A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450401998 (565h36m18.321485043s) [noise=true , address=45.91.116.5:4297 , piece_id=QUYLCQ7XKDFKUQU4KI3J5OB3ECMWBJ4TZ3RY7P2S4EE6K2XBW4SQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450974518 (565h36m18.322057814s) [noise=true , address=145.79.227.47:4267 , piece_id=XMN77OZAPUEB4NUYGJHAMTOW6GLEOE7G3BOAMWI3ZUDVNUWJYTTQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.451548161 (565h36m18.322631256s) [noise=true , address=77.90.31.17:4770 , piece_id=NPBLI7RMBP5BAOAYEBGEIH7YWRMFEOSNVPAVLSRTJSR6UQ66F4KA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452199420 (565h36m18.323282475s) [noise=true , address=145.79.227.15:4876 , piece_id=IMII2KVEAQMMLZUC5KE42YTRE37PI2JP7K4MMVUMNJVMEK4OS76Q, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452279543 (565h36m18.323362579s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452667662 (565h36m18.323750707s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.453060510 (565h36m18.324143555s) [noise=true , address=145.79.227.18:4007 , piece_id=M7RXQXZI4UJUQ6CMOEVFG3FHWOEGF6P4LN3NCVPOHWPO7Z65EWMA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.453574219 (565h36m18.324657264s) [noise=true , address=145.79.227.4:4185 , piece_id=X2LRCXCTDCSOXPJNZDD5SDLXRBU5MU6GLBX3DODHIVP3XFW5QBKQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:20.138758440 (565h36m20.009841475s) [upload=495898]: Next returned nil segment Dec 29 05:05:20.138775443 (565h36m20.009858488s) [upload=495898]: Waiting for error group managing segments... Dec 29 05:05:20.417061276 (565h36m20.288144321s) [noise=true , address=145.79.227.7:4295 , piece_id=RD56LWBXLXY7H7UC4QRRBAEIZ6GBO2LLUPAJDK7W42HQB77P2SDA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.438373497 (565h36m20.309456542s) [noise=true , address=145.79.227.46:4602 , piece_id=2WRL4WV3TUZWJ4QBCHPX3XS5A4HXMM6FMPDNR5KMQ7P3KC3CRK7A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.438983670 (565h36m20.310066715s) [noise=true , address=145.79.227.15:4876 , piece_id=IMII2KVEAQMMLZUC5KE42YTRE37PI2JP7K4MMVUMNJVMEK4OS76Q, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.439283922 (565h36m20.310366957s) [noise=true , address=145.79.227.39:4015 , piece_id=BLVDCPOGAE2MUQQMB6XDMA73KIDTZ7HYIKZJZ733BYEV3YFIDYHQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.440250381 (565h36m20.311333426s) [noise=true , address=145.79.227.11:4729 , piece_id=4QF6AFGC4O7KQIBEJBPX3SQFZGVXOSLAQBFIRY5X6UGYIXC6JCVQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.454050149 (565h36m20.325133184s) [noise=true , address=145.79.227.4:4185 , piece_id=X2LRCXCTDCSOXPJNZDD5SDLXRBU5MU6GLBX3DODHIVP3XFW5QBKQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.455700752 (565h36m20.326783797s) [noise=true , address=145.79.227.47:4267 , piece_id=XMN77OZAPUEB4NUYGJHAMTOW6GLEOE7G3BOAMWI3ZUDVNUWJYTTQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.459085748 (565h36m20.330168783s) [noise=true , address=145.79.227.26:4655 , piece_id=AD6TNQDFKZCXJRBEDOTBHRILASW2JM6QJHLA56ULAKWQPCDFGC3A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.459918964 (565h36m20.331001999s) [noise=true , address=145.79.227.18:4007 , piece_id=M7RXQXZI4UJUQ6CMOEVFG3FHWOEGF6P4LN3NCVPOHWPO7Z65EWMA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.704694136 (565h36m20.575777171s) [noise=true , address=77.90.31.14:4661 , piece_id=CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.791101300 (565h36m20.662184345s) [noise=true , address=45.91.116.12:4227 , piece_id=4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.837150746 (565h36m20.708233791s) [noise=true , address=77.90.31.17:4770 , piece_id=NPBLI7RMBP5BAOAYEBGEIH7YWRMFEOSNVPAVLSRTJSR6UQ66F4KA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.843066351 (565h36m20.714149386s) [noise=true , address=45.91.116.5:4297 , piece_id=QUYLCQ7XKDFKUQU4KI3J5OB3ECMWBJ4TZ3RY7P2S4EE6K2XBW4SQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.843089795 (565h36m20.71417284s) [seg_pos={0 0}, upload=495898]: Segment reached optimal threshold of 13 pieces. Dec 29 05:05:20.843263455 (565h36m20.71434649s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843269638 (565h36m20.714352673s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843538850 (565h36m20.714621885s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843545442 (565h36m20.714628477s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843651455 (565h36m20.7147345s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843657116 (565h36m20.714740161s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843673486 (565h36m20.714756521s) [seg_pos={0 0}, upload=495898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:05:20.843744972 (565h36m20.714828018s) [upload=495898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:05:20.843749932 (565h36m20.714832967s) [upload=495898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:05:21.143609134 (565h36m21.01469218s) [upload=495898]: Done uploading segments... Dec 29 05:05:21.143634652 (565h36m21.014717697s) [upload=495898]: Upload finished. err: <nil> Duration: 6.036828726s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.......X.....X..XX......X.X.....X.....X........X..X...X....] duration: 3.555388604s trace: https://jaeger.dev.storj.io/trace/3928df1d434145a9 FAIL (2025-12-29 05:05:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555439702s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XX.X.....XX.......X.XX..XX..XXX........XX......X...X..XXX] duration: 30.003167052s trace: https://jaeger.dev.storj.io/trace/17ba19948c5ed686 FAIL (2025-12-29 05:05:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ) context deadline exceeded ======LOGS====== Dec 29 05:05:18.181732997 (565h36m18.052816032s) [upload=495898]: Starting upload Dec 29 05:05:18.181790907 (565h36m18.052873942s) [upload=495898]: Done starting upload Dec 29 05:05:18.181950892 (565h36m18.053033937s) [upload=495898]: Uploading segments... Dec 29 05:05:18.182057124 (565h36m18.053140169s) [upload=495898]: Got next segment. Inline: false Dec 29 05:05:18.182102791 (565h36m18.053185826s) [upload=495898]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:05:18.182109564 (565h36m18.053192609s) [upload=495898]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:05:18.447800274 (565h36m18.318883329s) [seg_pos={0 0}, upload=495898]: Begin upload segment... Dec 29 05:05:18.447877992 (565h36m18.318961037s) [seg_pos={0 0}, upload=495898]: Done begin upload segment. Dec 29 05:05:18.447917717 (565h36m18.319000762s) [noise=true , address=145.79.227.39:4015 , piece_id=BLVDCPOGAE2MUQQMB6XDMA73KIDTZ7HYIKZJZ733BYEV3YFIDYHQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.448199444 (565h36m18.31928249s) [noise=true , address=145.79.227.11:4729 , piece_id=4QF6AFGC4O7KQIBEJBPX3SQFZGVXOSLAQBFIRY5X6UGYIXC6JCVQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.448758288 (565h36m18.319841333s) [noise=true , address=145.79.227.46:4602 , piece_id=2WRL4WV3TUZWJ4QBCHPX3XS5A4HXMM6FMPDNR5KMQ7P3KC3CRK7A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449100429 (565h36m18.320183464s) [noise=true , address=145.79.227.7:4295 , piece_id=RD56LWBXLXY7H7UC4QRRBAEIZ6GBO2LLUPAJDK7W42HQB77P2SDA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449440938 (565h36m18.320523983s) [noise=true , address=77.90.31.14:4661 , piece_id=CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.449781406 (565h36m18.320864451s) [noise=true , address=45.91.116.12:4227 , piece_id=4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450112496 (565h36m18.321195541s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450191407 (565h36m18.321274452s) [noise=true , address=145.79.227.26:4655 , piece_id=AD6TNQDFKZCXJRBEDOTBHRILASW2JM6QJHLA56ULAKWQPCDFGC3A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450401998 (565h36m18.321485043s) [noise=true , address=45.91.116.5:4297 , piece_id=QUYLCQ7XKDFKUQU4KI3J5OB3ECMWBJ4TZ3RY7P2S4EE6K2XBW4SQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.450974518 (565h36m18.322057814s) [noise=true , address=145.79.227.47:4267 , piece_id=XMN77OZAPUEB4NUYGJHAMTOW6GLEOE7G3BOAMWI3ZUDVNUWJYTTQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.451548161 (565h36m18.322631256s) [noise=true , address=77.90.31.17:4770 , piece_id=NPBLI7RMBP5BAOAYEBGEIH7YWRMFEOSNVPAVLSRTJSR6UQ66F4KA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452199420 (565h36m18.323282475s) [noise=true , address=145.79.227.15:4876 , piece_id=IMII2KVEAQMMLZUC5KE42YTRE37PI2JP7K4MMVUMNJVMEK4OS76Q, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452279543 (565h36m18.323362579s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.452667662 (565h36m18.323750707s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.453060510 (565h36m18.324143555s) [noise=true , address=145.79.227.18:4007 , piece_id=M7RXQXZI4UJUQ6CMOEVFG3FHWOEGF6P4LN3NCVPOHWPO7Z65EWMA, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:18.453574219 (565h36m18.324657264s) [noise=true , address=145.79.227.4:4185 , piece_id=X2LRCXCTDCSOXPJNZDD5SDLXRBU5MU6GLBX3DODHIVP3XFW5QBKQ, seg_pos={0 0}, upload=495898]: Uploading piece... Dec 29 05:05:20.138758440 (565h36m20.009841475s) [upload=495898]: Next returned nil segment Dec 29 05:05:20.138775443 (565h36m20.009858488s) [upload=495898]: Waiting for error group managing segments... Dec 29 05:05:20.417061276 (565h36m20.288144321s) [noise=true , address=145.79.227.7:4295 , piece_id=RD56LWBXLXY7H7UC4QRRBAEIZ6GBO2LLUPAJDK7W42HQB77P2SDA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.438373497 (565h36m20.309456542s) [noise=true , address=145.79.227.46:4602 , piece_id=2WRL4WV3TUZWJ4QBCHPX3XS5A4HXMM6FMPDNR5KMQ7P3KC3CRK7A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.438983670 (565h36m20.310066715s) [noise=true , address=145.79.227.15:4876 , piece_id=IMII2KVEAQMMLZUC5KE42YTRE37PI2JP7K4MMVUMNJVMEK4OS76Q, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.439283922 (565h36m20.310366957s) [noise=true , address=145.79.227.39:4015 , piece_id=BLVDCPOGAE2MUQQMB6XDMA73KIDTZ7HYIKZJZ733BYEV3YFIDYHQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.440250381 (565h36m20.311333426s) [noise=true , address=145.79.227.11:4729 , piece_id=4QF6AFGC4O7KQIBEJBPX3SQFZGVXOSLAQBFIRY5X6UGYIXC6JCVQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.454050149 (565h36m20.325133184s) [noise=true , address=145.79.227.4:4185 , piece_id=X2LRCXCTDCSOXPJNZDD5SDLXRBU5MU6GLBX3DODHIVP3XFW5QBKQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.455700752 (565h36m20.326783797s) [noise=true , address=145.79.227.47:4267 , piece_id=XMN77OZAPUEB4NUYGJHAMTOW6GLEOE7G3BOAMWI3ZUDVNUWJYTTQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.459085748 (565h36m20.330168783s) [noise=true , address=145.79.227.26:4655 , piece_id=AD6TNQDFKZCXJRBEDOTBHRILASW2JM6QJHLA56ULAKWQPCDFGC3A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.459918964 (565h36m20.331001999s) [noise=true , address=145.79.227.18:4007 , piece_id=M7RXQXZI4UJUQ6CMOEVFG3FHWOEGF6P4LN3NCVPOHWPO7Z65EWMA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.704694136 (565h36m20.575777171s) [noise=true , address=77.90.31.14:4661 , piece_id=CWCNS5LCI72K2TGCBADOA2YX43SVBMXECUVO4MNCCVKJUA55XIIA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.791101300 (565h36m20.662184345s) [noise=true , address=45.91.116.12:4227 , piece_id=4CVQOGHOV2MTV3HNNX7ZRPVQ3736OSIN2YADF4GQQ63JQZ2Y3OAQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.837150746 (565h36m20.708233791s) [noise=true , address=77.90.31.17:4770 , piece_id=NPBLI7RMBP5BAOAYEBGEIH7YWRMFEOSNVPAVLSRTJSR6UQ66F4KA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.843066351 (565h36m20.714149386s) [noise=true , address=45.91.116.5:4297 , piece_id=QUYLCQ7XKDFKUQU4KI3J5OB3ECMWBJ4TZ3RY7P2S4EE6K2XBW4SQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: <nil> Dec 29 05:05:20.843089795 (565h36m20.71417284s) [seg_pos={0 0}, upload=495898]: Segment reached optimal threshold of 13 pieces. Dec 29 05:05:20.843263455 (565h36m20.71434649s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843269638 (565h36m20.714352673s) [noise=true , address=45.91.116.4:4939 , piece_id=5BLLXBKUAAKWYVGR43RVOSJEUF4FHEUXMOTSYIQEH2XRN6RTDNEA, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843538850 (565h36m20.714621885s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843545442 (565h36m20.714628477s) [noise=true , address=77.90.31.18:4805 , piece_id=LEV252DC22SPVH4VMWV2LPURZYPERFUHUPAGDW75K3FBX6VAAQ3A, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843651455 (565h36m20.7147345s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 29 05:05:20.843657116 (565h36m20.714740161s) [noise=true , address=77.90.31.11:4556 , piece_id=TSMDYZPO6H4HCPKQDIB4LV64YSLIZFARK2KGPYIHA2NGI5N5QFDQ, seg_pos={0 0}, upload=495898]: Long tail cancellation due to optimal threshold Dec 29 05:05:20.843673486 (565h36m20.714756521s) [seg_pos={0 0}, upload=495898]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:05:20.843744972 (565h36m20.714828018s) [upload=495898]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:05:20.843749932 (565h36m20.714832967s) [upload=495898]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:05:21.143609134 (565h36m21.01469218s) [upload=495898]: Done uploading segments... Dec 29 05:05:21.143634652 (565h36m21.014717697s) [upload=495898]: Upload finished. err: <nil> Duration: 30.003191118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [XX.......X.....X..XX......X.X.....X.....X........X..X...X...] duration: 4.711801865s trace: https://jaeger.dev.storj.io/trace/3928df1d434145a9 FAIL (2025-12-29 05:05:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.711826342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..XX.X.....XX.......X.XX..XX..XXX........XX......X...X..XX] duration: 30.002099792s trace: https://jaeger.dev.storj.io/trace/5c0958ed319e581a FAIL (2025-12-29 05:05:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: IXZRYCGZ6TF4FZKEKYU6LQ2MJDCZM2UX53CNESF2DYACXBAOZ5BQ) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: IFILHOLDLFQKLXMYS77ZZOBEYRV654Q664RPQDZVIDIJPOEE5J4Q) context deadline exceeded ======LOGS====== Dec 29 05:04:47.175360463 (565h35m47.046443498s) [upload=495890]: Starting upload Dec 29 05:04:47.175421197 (565h35m47.046504242s) [upload=495890]: Done starting upload Dec 29 05:04:47.175560833 (565h35m47.046643868s) [upload=495890]: Uploading segments... Dec 29 05:04:47.175653460 (565h35m47.046736495s) [upload=495890]: Got next segment. Inline: false Dec 29 05:04:47.175686763 (565h35m47.046769799s) [upload=495890]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:04:47.175691572 (565h35m47.046774608s) [upload=495890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:04:47.420423844 (565h35m47.291506879s) [seg_pos={0 0}, upload=495890]: Begin upload segment... Dec 29 05:04:47.420483818 (565h35m47.291566853s) [seg_pos={0 0}, upload=495890]: Done begin upload segment. Dec 29 05:04:47.420557077 (565h35m47.291640123s) [noise=true , address=145.79.227.25:4617 , piece_id=46GRAA7SYB7XKLV7PCICPNDWGRHY6FAE2LIYAT6KQC42HFGQDWQA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.420572557 (565h35m47.291655592s) [noise=true , address=77.90.31.21:4877 , piece_id=IXZRYCGZ6TF4FZKEKYU6LQ2MJDCZM2UX53CNESF2DYACXBAOZ5BQ, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.421104179 (565h35m47.292187224s) [noise=true , address=145.79.227.45:4231 , piece_id=CK7DLY7IZEOV6XMG2N3TIORC5566L7UXKRDMGT47EKU2TGTGCQPQ, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.421475416 (565h35m47.292558451s) [noise=true , address=77.90.31.11:4553 , piece_id=ZEWVS4336HQJ7G5M5X652S34K5LMILOD3PLVIYRPEWLXDNMF7EDA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.422138548 (565h35m47.293221584s) [noise=true , address=145.79.227.37:4566 , piece_id=SMF3RFFR2LMPRURRXS4XJM5M4XT2UMC2GDTC7NKDSVU7X2AK2RGA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.422553429 (565h35m47.293636474s) [noise=true , address=145.79.227.5:4223 , piece_id=H3Z4PLQFGEJENA6BJP6TI6IFHXF4UQRGH6A77MSOT2U4G3JEUSRA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.423150946 (565h35m47.294233991s) [noise=true , address=145.79.227.17:4970 , piece_id=AQBX7LXCESFPHCZZZTQZNWM4N3GGOBBLX44L3ZB2DFVXVJX46ZJA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.423578962 (565h35m47.294661997s) [noise=true , address=145.79.227.15:4876 , piece_id=CVCAOC3ZA7UIQECWP2ITTVPB4NMF5Y75HS6OP4QRLLDTPIN5VIKQ, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.425397494 (565h35m47.296480529s) [noise=true , address=45.91.116.12:4228 , piece_id=2W4DEN6MFO5NJ7KFZF4KR6QLNCRFVDLCOXMDSQXHVLYKMLIJ6JSA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.427257786 (565h35m47.298340821s) [noise=true , address=145.79.227.11:4732 , piece_id=TNNT5IEHBBVL5HROSJINXIH4G3LCAOEOBUUEXM4KAJLVDATCOFGA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.429141784 (565h35m47.300224819s) [noise=true , address=145.79.227.26:4655 , piece_id=CICRQTLE52UZOWNK6U7H6NRG3LJ7SKENUWTAJMRJ3T24XY4QMXZQ, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.431366600 (565h35m47.302449635s) [noise=true , address=45.91.116.13:4155 , piece_id=UZYUT7XQLYYQHLL7AAJITIELOVCS4OML2X6ZMQCAIGEP3ZDQEB5Q, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.431903493 (565h35m47.302986538s) [noise=true , address=45.91.116.9:4261 , piece_id=IFILHOLDLFQKLXMYS77ZZOBEYRV654Q664RPQDZVIDIJPOEE5J4Q, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.433330831 (565h35m47.304413866s) [noise=true , address=145.79.227.16:4911 , piece_id=ZP2Q3TIV44XNYTHANTIIB7SLYPAFZMC43DD2WO7CGOSADX32LEOA, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.433443906 (565h35m47.304526951s) [noise=true , address=145.79.227.46:4602 , piece_id=MTMNPIF5RTFMCMNQW4CAE6CX4DQQPMTIH46UOYNC4AT7IJNXP26A, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:47.434005676 (565h35m47.305088711s) [noise=true , address=45.91.116.4:4939 , piece_id=UEGULZVVOWSFGSKP3DWEED5FJYY3SHLQEFDDAAJPTHJZFDVYZ3AQ, seg_pos={0 0}, upload=495890]: Uploading piece... Dec 29 05:04:49.114276702 (565h35m48.985359737s) [upload=495890]: Next returned nil segment Dec 29 05:04:49.114289887 (565h35m48.985372932s) [upload=495890]: Waiting for error group managing segments... Dec 29 05:04:49.400564992 (565h35m49.271648037s) [noise=true , address=145.79.227.5:4223 , piece_id=H3Z4PLQFGEJENA6BJP6TI6IFHXF4UQRGH6A77MSOT2U4G3JEUSRA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.403540137 (565h35m49.274623182s) [noise=true , address=145.79.227.11:4732 , piece_id=TNNT5IEHBBVL5HROSJINXIH4G3LCAOEOBUUEXM4KAJLVDATCOFGA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.406488271 (565h35m49.277571306s) [noise=true , address=145.79.227.25:4617 , piece_id=46GRAA7SYB7XKLV7PCICPNDWGRHY6FAE2LIYAT6KQC42HFGQDWQA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.410299378 (565h35m49.281382414s) [noise=true , address=145.79.227.46:4602 , piece_id=MTMNPIF5RTFMCMNQW4CAE6CX4DQQPMTIH46UOYNC4AT7IJNXP26A, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.414038568 (565h35m49.285121603s) [noise=true , address=145.79.227.37:4566 , piece_id=SMF3RFFR2LMPRURRXS4XJM5M4XT2UMC2GDTC7NKDSVU7X2AK2RGA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.415734788 (565h35m49.286817833s) [noise=true , address=145.79.227.17:4970 , piece_id=AQBX7LXCESFPHCZZZTQZNWM4N3GGOBBLX44L3ZB2DFVXVJX46ZJA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.427106903 (565h35m49.298189938s) [noise=true , address=145.79.227.45:4231 , piece_id=CK7DLY7IZEOV6XMG2N3TIORC5566L7UXKRDMGT47EKU2TGTGCQPQ, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.435190505 (565h35m49.306273541s) [noise=true , address=145.79.227.26:4655 , piece_id=CICRQTLE52UZOWNK6U7H6NRG3LJ7SKENUWTAJMRJ3T24XY4QMXZQ, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.435668365 (565h35m49.3067514s) [noise=true , address=145.79.227.15:4876 , piece_id=CVCAOC3ZA7UIQECWP2ITTVPB4NMF5Y75HS6OP4QRLLDTPIN5VIKQ, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.444451600 (565h35m49.315534635s) [noise=true , address=145.79.227.16:4911 , piece_id=ZP2Q3TIV44XNYTHANTIIB7SLYPAFZMC43DD2WO7CGOSADX32LEOA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.777867990 (565h35m49.648951035s) [noise=true , address=77.90.31.21:4877 , piece_id=IXZRYCGZ6TF4FZKEKYU6LQ2MJDCZM2UX53CNESF2DYACXBAOZ5BQ, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:49.797346250 (565h35m49.668429285s) [noise=true , address=45.91.116.4:4939 , piece_id=UEGULZVVOWSFGSKP3DWEED5FJYY3SHLQEFDDAAJPTHJZFDVYZ3AQ, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:50.166891726 (565h35m50.037974762s) [noise=true , address=45.91.116.9:4261 , piece_id=IFILHOLDLFQKLXMYS77ZZOBEYRV654Q664RPQDZVIDIJPOEE5J4Q, seg_pos={0 0}, upload=495890]: Done uploading piece. err: <nil> Dec 29 05:04:50.166914900 (565h35m50.037997945s) [seg_pos={0 0}, upload=495890]: Segment reached optimal threshold of 13 pieces. Dec 29 05:04:50.167145149 (565h35m50.038228194s) [noise=true , address=45.91.116.12:4228 , piece_id=2W4DEN6MFO5NJ7KFZF4KR6QLNCRFVDLCOXMDSQXHVLYKMLIJ6JSA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 05:04:50.167155709 (565h35m50.038238754s) [noise=true , address=45.91.116.12:4228 , piece_id=2W4DEN6MFO5NJ7KFZF4KR6QLNCRFVDLCOXMDSQXHVLYKMLIJ6JSA, seg_pos={0 0}, upload=495890]: Long tail cancellation due to optimal threshold Dec 29 05:04:50.167180206 (565h35m50.038263241s) [noise=true , address=45.91.116.13:4155 , piece_id=UZYUT7XQLYYQHLL7AAJITIELOVCS4OML2X6ZMQCAIGEP3ZDQEB5Q, seg_pos={0 0}, upload=495890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 05:04:50.167190395 (565h35m50.03827343s) [noise=true , address=45.91.116.13:4155 , piece_id=UZYUT7XQLYYQHLL7AAJITIELOVCS4OML2X6ZMQCAIGEP3ZDQEB5Q, seg_pos={0 0}, upload=495890]: Long tail cancellation due to optimal threshold Dec 29 05:04:50.167353746 (565h35m50.038436801s) [noise=true , address=77.90.31.11:4553 , piece_id=ZEWVS4336HQJ7G5M5X652S34K5LMILOD3PLVIYRPEWLXDNMF7EDA, seg_pos={0 0}, upload=495890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 05:04:50.167367341 (565h35m50.038450387s) [noise=true , address=77.90.31.11:4553 , piece_id=ZEWVS4336HQJ7G5M5X652S34K5LMILOD3PLVIYRPEWLXDNMF7EDA, seg_pos={0 0}, upload=495890]: Long tail cancellation due to optimal threshold Dec 29 05:04:50.167390065 (565h35m50.03847311s) [seg_pos={0 0}, upload=495890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:04:50.167478113 (565h35m50.038561158s) [upload=495890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:04:50.167485457 (565h35m50.038568502s) [upload=495890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:04:50.457999423 (565h35m50.329082458s) [upload=495890]: Done uploading segments... Dec 29 05:04:50.458017368 (565h35m50.329100403s) [upload=495890]: Upload finished. err: <nil> Duration: 30.002110222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X...X..X] duration: 30.005156658s trace: https://jaeger.dev.storj.io/trace/471007ee3de4bdfa FAIL (2025-12-29 05:04:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (565h35m16.808804012s) [upload=495882]: Done starting upload Dec 29 05:04:16.937869951 (565h35m16.808952996s) [upload=495882]: Uploading segments... Dec 29 05:04:16.937984179 (565h35m16.809067224s) [upload=495882]: Got next segment. Inline: false Dec 29 05:04:16.938031218 (565h35m16.809114253s) [upload=495882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:04:16.938037319 (565h35m16.809120354s) [upload=495882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:04:17.186789746 (565h35m17.057872791s) [seg_pos={0 0}, upload=495882]: Begin upload segment... Dec 29 05:04:17.186844840 (565h35m17.057927886s) [seg_pos={0 0}, upload=495882]: Done begin upload segment. Dec 29 05:04:17.186864068 (565h35m17.057947113s) [noise=true , address=145.79.227.41:4087 , piece_id=24OIMLGECPLZTXIRB67PPIFRZZWLP6VVRIRDQZ4DH4QJEWNHY6IA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.186884836 (565h35m17.057967872s) [noise=true , address=77.90.31.10:4517 , piece_id=IHFCJJHVXXUBEA5WJAU4QCTTNHIOGZUCFPKAW2FOHQZC4DK5BHYQ, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.187378447 (565h35m17.058461482s) [noise=true , address=77.90.31.18:4805 , piece_id=W2FEFCQT5AD5RLNTEDZNIMSHUZNUHT6JOBKGM5MAHFUQPM7NO2XA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.187696503 (565h35m17.058779538s) [noise=true , address=145.79.227.2:4114 , piece_id=WWA3W5PYAPWGOXJF5P6ZFO67QX3WCTCN4LYULN3MHCIXMIPSURYA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.188023675 (565h35m17.05910672s) [noise=true , address=145.79.227.50:4374 , piece_id=6FZCVNTIW7EPFZWNSEZSYH6MSWJ2G7MBEISWESPBF4Y64Z2EYXAA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.188867171 (565h35m17.059950216s) [noise=true , address=145.79.227.21:4473 , piece_id=N6WS3XJZMLTZHCA4A27DF663ALLX6DCHTJJBOCFXJEHEFPHBVYXQ, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.189187180 (565h35m17.060270225s) [noise=true , address=145.79.227.14:4838 , piece_id=6HETZICP2575F7SD5ERSISYJ26UGWOC2MQDWLPE2XGF6NWI34MYA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.189495738 (565h35m17.060578773s) [noise=true , address=145.79.227.46:4602 , piece_id=EFEQRTTIZDEUAZZOZA7RQWQBYTQHSZ3WW7LIG6MMABCI3FV2XXHA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.189880020 (565h35m17.060963065s) [noise=true , address=145.79.227.24:4582 , piece_id=DGVVUNDM2XSZ6GTN4ANC3HWP3XQL25E7762A2RB34D7MMTQUMR7Q, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.190797157 (565h35m17.061880192s) [noise=true , address=45.91.116.12:4226 , piece_id=C3W2TRCW2L53QNBDRV4TPEOT63URMUPYAUPTHXH6QOL4W4U5APBA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.191129649 (565h35m17.062212694s) [noise=true , address=45.91.116.9:4263 , piece_id=MYXPVD2GAHIOX773QKDYOZJY6RVKHFD7NC75RFDN3FOV2R2M6AGA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.192416740 (565h35m17.063499785s) [noise=true , address=145.79.227.17:4971 , piece_id=GCHI5E5ZZZV52OO3ZSEMSZOWN2LM5PWNASXA2OB3MM4M6UOWUTDA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.193794634 (565h35m17.064877669s) [noise=true , address=145.79.227.15:4875 , piece_id=7TJSY4GSHRKMYE6XK3NZPBMAS67ETIVAXK6DZ2WKCXBTO3HUIGPQ, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.195742694 (565h35m17.066825739s) [noise=true , address=145.79.227.16:4911 , piece_id=AGTK5AS7EO4YGM5JCKT4GHAWCHGAXN2K6IDJNIMZ7GBHMANVFNSQ, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.199703917 (565h35m17.070786952s) [noise=true , address=145.79.227.47:4266 , piece_id=LWPZPBFBUD34MYXPKUJF4XUUSSRMQ4D35F3JTORP32VJOX2WLEJA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:17.202735439 (565h35m17.073818484s) [noise=true , address=45.91.116.4:4939 , piece_id=A2GQIX4W46REDI6AE735MTBZ2WLXWHSRMWQT3B5UKZ64RNDVWMXA, seg_pos={0 0}, upload=495882]: Uploading piece... Dec 29 05:04:18.887573451 (565h35m18.758656486s) [upload=495882]: Next returned nil segment Dec 29 05:04:18.887586577 (565h35m18.758669622s) [upload=495882]: Waiting for error group managing segments... Dec 29 05:04:19.171145529 (565h35m19.042228564s) [noise=true , address=145.79.227.2:4114 , piece_id=WWA3W5PYAPWGOXJF5P6ZFO67QX3WCTCN4LYULN3MHCIXMIPSURYA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.174501129 (565h35m19.045584174s) [noise=true , address=145.79.227.41:4087 , piece_id=24OIMLGECPLZTXIRB67PPIFRZZWLP6VVRIRDQZ4DH4QJEWNHY6IA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.174532669 (565h35m19.045615714s) [noise=true , address=145.79.227.50:4374 , piece_id=6FZCVNTIW7EPFZWNSEZSYH6MSWJ2G7MBEISWESPBF4Y64Z2EYXAA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.189893639 (565h35m19.060976674s) [noise=true , address=145.79.227.21:4473 , piece_id=N6WS3XJZMLTZHCA4A27DF663ALLX6DCHTJJBOCFXJEHEFPHBVYXQ, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.195452655 (565h35m19.0665357s) [noise=true , address=145.79.227.46:4602 , piece_id=EFEQRTTIZDEUAZZOZA7RQWQBYTQHSZ3WW7LIG6MMABCI3FV2XXHA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.195709323 (565h35m19.066792368s) [noise=true , address=145.79.227.24:4582 , piece_id=DGVVUNDM2XSZ6GTN4ANC3HWP3XQL25E7762A2RB34D7MMTQUMR7Q, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.209758796 (565h35m19.080841841s) [noise=true , address=145.79.227.47:4266 , piece_id=LWPZPBFBUD34MYXPKUJF4XUUSSRMQ4D35F3JTORP32VJOX2WLEJA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.210786022 (565h35m19.081869067s) [noise=true , address=145.79.227.17:4971 , piece_id=GCHI5E5ZZZV52OO3ZSEMSZOWN2LM5PWNASXA2OB3MM4M6UOWUTDA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.212394073 (565h35m19.083477118s) [noise=true , address=145.79.227.14:4838 , piece_id=6HETZICP2575F7SD5ERSISYJ26UGWOC2MQDWLPE2XGF6NWI34MYA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.215011477 (565h35m19.086094522s) [noise=true , address=145.79.227.16:4911 , piece_id=AGTK5AS7EO4YGM5JCKT4GHAWCHGAXN2K6IDJNIMZ7GBHMANVFNSQ, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.216175795 (565h35m19.08725884s) [noise=true , address=145.79.227.15:4875 , piece_id=7TJSY4GSHRKMYE6XK3NZPBMAS67ETIVAXK6DZ2WKCXBTO3HUIGPQ, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.380169566 (565h35m19.251252631s) [noise=true , address=77.90.31.10:4517 , piece_id=IHFCJJHVXXUBEA5WJAU4QCTTNHIOGZUCFPKAW2FOHQZC4DK5BHYQ, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.501502494 (565h35m19.372585539s) [noise=true , address=45.91.116.12:4226 , piece_id=C3W2TRCW2L53QNBDRV4TPEOT63URMUPYAUPTHXH6QOL4W4U5APBA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: <nil> Dec 29 05:04:19.501526399 (565h35m19.372609445s) [seg_pos={0 0}, upload=495882]: Segment reached optimal threshold of 13 pieces. Dec 29 05:04:19.501863442 (565h35m19.372946487s) [noise=true , address=45.91.116.4:4939 , piece_id=A2GQIX4W46REDI6AE735MTBZ2WLXWHSRMWQT3B5UKZ64RNDVWMXA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: read tcp 188.166.225.170:41438->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 05:04:19.501873120 (565h35m19.372956165s) [noise=true , address=45.91.116.9:4263 , piece_id=MYXPVD2GAHIOX773QKDYOZJY6RVKHFD7NC75RFDN3FOV2R2M6AGA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 29 05:04:19.501887057 (565h35m19.372970102s) [noise=true , address=45.91.116.9:4263 , piece_id=MYXPVD2GAHIOX773QKDYOZJY6RVKHFD7NC75RFDN3FOV2R2M6AGA, seg_pos={0 0}, upload=495882]: Long tail cancellation due to optimal threshold Dec 29 05:04:19.501901133 (565h35m19.372984178s) [noise=true , address=45.91.116.4:4939 , piece_id=A2GQIX4W46REDI6AE735MTBZ2WLXWHSRMWQT3B5UKZ64RNDVWMXA, seg_pos={0 0}, upload=495882]: Long tail cancellation due to optimal threshold Dec 29 05:04:19.502009329 (565h35m19.373092364s) [noise=true , address=77.90.31.18:4805 , piece_id=W2FEFCQT5AD5RLNTEDZNIMSHUZNUHT6JOBKGM5MAHFUQPM7NO2XA, seg_pos={0 0}, upload=495882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:41284->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:41284->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41284->77.90.31.18:4805: use of closed network connection Dec 29 05:04:19.502018126 (565h35m19.373101171s) [noise=true , address=77.90.31.18:4805 , piece_id=W2FEFCQT5AD5RLNTEDZNIMSHUZNUHT6JOBKGM5MAHFUQPM7NO2XA, seg_pos={0 0}, upload=495882]: Long tail cancellation due to optimal threshold Dec 29 05:04:19.502131322 (565h35m19.373214367s) [seg_pos={0 0}, upload=495882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:04:19.502219660 (565h35m19.373302695s) [upload=495882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:04:19.502226342 (565h35m19.373309387s) [upload=495882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:04:19.790837325 (565h35m19.661920371s) [upload=495882]: Done uploading segments... Dec 29 05:04:19.790860891 (565h35m19.661943936s) [upload=495882]: Upload finished. err: <nil> Duration: 30.005176446s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.XX.......X.....X..XX......X.X.....X.....X........X..X...X..] duration: 3.458890354s trace: https://jaeger.dev.storj.io/trace/3928df1d434145a9 FAIL (2025-12-29 05:04:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.458920582s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..XX.......X.....X..XX......X.X.....X.....X........X..X...X.] duration: 3.939745262s trace: https://jaeger.dev.storj.io/trace/3928df1d434145a9 FAIL (2025-12-29 05:04:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.939795969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X...X..] duration: 6.035757049s trace: https://jaeger.dev.storj.io/trace/2d957f66117cb224 FAIL (2025-12-29 05:03:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:02:03.695864504 (565h33m3.56694754s) [upload=495851]: Starting upload Dec 29 05:02:03.695969835 (565h33m3.56705288s) [upload=495851]: Done starting upload Dec 29 05:02:03.696142474 (565h33m3.567225519s) [upload=495851]: Uploading segments... Dec 29 05:02:03.696247293 (565h33m3.567330348s) [upload=495851]: Got next segment. Inline: false Dec 29 05:02:03.696294584 (565h33m3.567377629s) [upload=495851]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:02:03.696302739 (565h33m3.567385784s) [upload=495851]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:02:03.951067754 (565h33m3.822150799s) [seg_pos={0 0}, upload=495851]: Begin upload segment... Dec 29 05:02:03.951150241 (565h33m3.822233286s) [seg_pos={0 0}, upload=495851]: Done begin upload segment. Dec 29 05:02:03.951215325 (565h33m3.82229837s) [noise=true , address=45.91.116.11:4117 , piece_id=POSJ3MZUV2LHSZIDWTRS74H7TGEFLXINT3MECMLJ63IWDRZDRNBA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.951224143 (565h33m3.822307178s) [noise=true , address=45.91.116.5:4297 , piece_id=KSHKJABPELNNJM3PI452PEHTLD4NP4HFXNJOKAQKSUFLEVZGMFJQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.951739273 (565h33m3.822822308s) [noise=true , address=77.90.31.21:4878 , piece_id=IHR4NYH2ITJOOMBLMTV3LZKUH4FYAREZVJV2ETZR3VIZCQK455UA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952038814 (565h33m3.823121859s) [noise=true , address=145.79.227.11:4730 , piece_id=6RFVK2UTGIZYYPWNYIGJ32HIDA5FJBWHSTUWCJOM2B6ZJ4TCIVJQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952399821 (565h33m3.823482866s) [noise=true , address=145.79.227.4:4185 , piece_id=YIWQKK6UWQYLUEZYM4RWYREFHVYGTVBTGNKT3OHNAGIZ63V7ZDOA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952817827 (565h33m3.823900862s) [noise=true , address=145.79.227.17:4972 , piece_id=FNQJGEWU4WJU6PQLOGESRX62CC6GW3AZ774EPRLMRIAG4YHP6HFA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.953148727 (565h33m3.824231762s) [noise=true , address=145.79.227.41:4085 , piece_id=3XZ3M44UQWUJQVNFQ7U3UUMKCWP5F3LP45VC5RDCLR7POBL7D55A, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.954432772 (565h33m3.825515817s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955051461 (565h33m3.826134506s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955496788 (565h33m3.826579833s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955940753 (565h33m3.827023798s) [noise=true , address=145.79.227.32:4835 , piece_id=EWTEJ6COP33NYV3KJ6G33VHE74NWUUVW5CUNGPPKSW2WOT2NLUNQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.956460483 (565h33m3.827543528s) [noise=true , address=145.79.227.38:4977 , piece_id=L6B3W4NBKZD2W7GGYATATRXF74F2G7EYR6U33RLRBKTYGVGEZEBQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.956931761 (565h33m3.828014806s) [noise=true , address=145.79.227.51:4411 , piece_id=N2NMHY2ARKS6VF36ESHBVZZZ5WQOFZL7B54GZOYTI3XO4JSZAYNA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.957418147 (565h33m3.828501192s) [noise=true , address=145.79.227.15:4873 , piece_id=VN3CYY7J7FXQMZFWP356DBEJONINKOXAIAUMCPLFWDT3JOFXVN2Q, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.957975589 (565h33m3.829058634s) [noise=true , address=145.79.227.37:4566 , piece_id=6MLSDZPOLCKKE2L554QYOIXPFIBQ3MVAEYEJ3NP3XE4UKEVDM3UQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.958731778 (565h33m3.829814843s) [noise=true , address=145.79.227.16:4912 , piece_id=OGXZZQ4S2QWEA5VS6B2SQCMPOPZU6D3B2DHCDSVD7K6Q4QHFX44A, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:05.650766284 (565h33m5.521849329s) [upload=495851]: Next returned nil segment Dec 29 05:02:05.650794408 (565h33m5.521877453s) [upload=495851]: Waiting for error group managing segments... Dec 29 05:02:05.930115341 (565h33m5.801198376s) [noise=true , address=145.79.227.4:4185 , piece_id=YIWQKK6UWQYLUEZYM4RWYREFHVYGTVBTGNKT3OHNAGIZ63V7ZDOA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.936825428 (565h33m5.807908463s) [noise=true , address=145.79.227.17:4972 , piece_id=FNQJGEWU4WJU6PQLOGESRX62CC6GW3AZ774EPRLMRIAG4YHP6HFA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.951504720 (565h33m5.822587775s) [noise=true , address=145.79.227.51:4411 , piece_id=N2NMHY2ARKS6VF36ESHBVZZZ5WQOFZL7B54GZOYTI3XO4JSZAYNA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.952360490 (565h33m5.823443535s) [noise=true , address=145.79.227.41:4085 , piece_id=3XZ3M44UQWUJQVNFQ7U3UUMKCWP5F3LP45VC5RDCLR7POBL7D55A, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.953443091 (565h33m5.824526126s) [noise=true , address=145.79.227.38:4977 , piece_id=L6B3W4NBKZD2W7GGYATATRXF74F2G7EYR6U33RLRBKTYGVGEZEBQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.954147141 (565h33m5.825230196s) [noise=true , address=145.79.227.16:4912 , piece_id=OGXZZQ4S2QWEA5VS6B2SQCMPOPZU6D3B2DHCDSVD7K6Q4QHFX44A, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.956890145 (565h33m5.82797319s) [noise=true , address=145.79.227.15:4873 , piece_id=VN3CYY7J7FXQMZFWP356DBEJONINKOXAIAUMCPLFWDT3JOFXVN2Q, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.967715939 (565h33m5.838798974s) [noise=true , address=145.79.227.11:4730 , piece_id=6RFVK2UTGIZYYPWNYIGJ32HIDA5FJBWHSTUWCJOM2B6ZJ4TCIVJQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.973899664 (565h33m5.844982709s) [noise=true , address=145.79.227.32:4835 , piece_id=EWTEJ6COP33NYV3KJ6G33VHE74NWUUVW5CUNGPPKSW2WOT2NLUNQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.997147833 (565h33m5.868230878s) [noise=true , address=145.79.227.37:4566 , piece_id=6MLSDZPOLCKKE2L554QYOIXPFIBQ3MVAEYEJ3NP3XE4UKEVDM3UQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.237297548 (565h33m6.108380603s) [noise=true , address=45.91.116.11:4117 , piece_id=POSJ3MZUV2LHSZIDWTRS74H7TGEFLXINT3MECMLJ63IWDRZDRNBA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.249321813 (565h33m6.120404849s) [noise=true , address=45.91.116.5:4297 , piece_id=KSHKJABPELNNJM3PI452PEHTLD4NP4HFXNJOKAQKSUFLEVZGMFJQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.333711816 (565h33m6.204794861s) [noise=true , address=77.90.31.21:4878 , piece_id=IHR4NYH2ITJOOMBLMTV3LZKUH4FYAREZVJV2ETZR3VIZCQK455UA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.333738558 (565h33m6.204821603s) [seg_pos={0 0}, upload=495851]: Segment reached optimal threshold of 13 pieces. Dec 29 05:02:06.334180178 (565h33m6.205263223s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334189266 (565h33m6.205272311s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334409936 (565h33m6.205492981s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334417921 (565h33m6.205500976s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334913925 (565h33m6.20599697s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334922833 (565h33m6.206005888s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334947569 (565h33m6.206030604s) [seg_pos={0 0}, upload=495851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:02:06.335035306 (565h33m6.206118352s) [upload=495851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:02:06.335042801 (565h33m6.206125836s) [upload=495851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:02:06.631244788 (565h33m6.502327823s) [upload=495851]: Done uploading segments... Dec 29 05:02:06.631265338 (565h33m6.502348383s) [upload=495851]: Upload finished. err: <nil> Duration: 6.035775715s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...XX.......X.....X..XX......X.X.....X.....X........X..X...X] duration: 5.001595942s trace: https://jaeger.dev.storj.io/trace/3928df1d434145a9 FAIL (2025-12-29 05:03:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001614006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X...X.] duration: 6.183509553s trace: https://jaeger.dev.storj.io/trace/2d957f66117cb224 FAIL (2025-12-29 05:02:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:02:03.695864504 (565h33m3.56694754s) [upload=495851]: Starting upload Dec 29 05:02:03.695969835 (565h33m3.56705288s) [upload=495851]: Done starting upload Dec 29 05:02:03.696142474 (565h33m3.567225519s) [upload=495851]: Uploading segments... Dec 29 05:02:03.696247293 (565h33m3.567330348s) [upload=495851]: Got next segment. Inline: false Dec 29 05:02:03.696294584 (565h33m3.567377629s) [upload=495851]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:02:03.696302739 (565h33m3.567385784s) [upload=495851]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:02:03.951067754 (565h33m3.822150799s) [seg_pos={0 0}, upload=495851]: Begin upload segment... Dec 29 05:02:03.951150241 (565h33m3.822233286s) [seg_pos={0 0}, upload=495851]: Done begin upload segment. Dec 29 05:02:03.951215325 (565h33m3.82229837s) [noise=true , address=45.91.116.11:4117 , piece_id=POSJ3MZUV2LHSZIDWTRS74H7TGEFLXINT3MECMLJ63IWDRZDRNBA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.951224143 (565h33m3.822307178s) [noise=true , address=45.91.116.5:4297 , piece_id=KSHKJABPELNNJM3PI452PEHTLD4NP4HFXNJOKAQKSUFLEVZGMFJQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.951739273 (565h33m3.822822308s) [noise=true , address=77.90.31.21:4878 , piece_id=IHR4NYH2ITJOOMBLMTV3LZKUH4FYAREZVJV2ETZR3VIZCQK455UA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952038814 (565h33m3.823121859s) [noise=true , address=145.79.227.11:4730 , piece_id=6RFVK2UTGIZYYPWNYIGJ32HIDA5FJBWHSTUWCJOM2B6ZJ4TCIVJQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952399821 (565h33m3.823482866s) [noise=true , address=145.79.227.4:4185 , piece_id=YIWQKK6UWQYLUEZYM4RWYREFHVYGTVBTGNKT3OHNAGIZ63V7ZDOA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952817827 (565h33m3.823900862s) [noise=true , address=145.79.227.17:4972 , piece_id=FNQJGEWU4WJU6PQLOGESRX62CC6GW3AZ774EPRLMRIAG4YHP6HFA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.953148727 (565h33m3.824231762s) [noise=true , address=145.79.227.41:4085 , piece_id=3XZ3M44UQWUJQVNFQ7U3UUMKCWP5F3LP45VC5RDCLR7POBL7D55A, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.954432772 (565h33m3.825515817s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955051461 (565h33m3.826134506s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955496788 (565h33m3.826579833s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955940753 (565h33m3.827023798s) [noise=true , address=145.79.227.32:4835 , piece_id=EWTEJ6COP33NYV3KJ6G33VHE74NWUUVW5CUNGPPKSW2WOT2NLUNQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.956460483 (565h33m3.827543528s) [noise=true , address=145.79.227.38:4977 , piece_id=L6B3W4NBKZD2W7GGYATATRXF74F2G7EYR6U33RLRBKTYGVGEZEBQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.956931761 (565h33m3.828014806s) [noise=true , address=145.79.227.51:4411 , piece_id=N2NMHY2ARKS6VF36ESHBVZZZ5WQOFZL7B54GZOYTI3XO4JSZAYNA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.957418147 (565h33m3.828501192s) [noise=true , address=145.79.227.15:4873 , piece_id=VN3CYY7J7FXQMZFWP356DBEJONINKOXAIAUMCPLFWDT3JOFXVN2Q, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.957975589 (565h33m3.829058634s) [noise=true , address=145.79.227.37:4566 , piece_id=6MLSDZPOLCKKE2L554QYOIXPFIBQ3MVAEYEJ3NP3XE4UKEVDM3UQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.958731778 (565h33m3.829814843s) [noise=true , address=145.79.227.16:4912 , piece_id=OGXZZQ4S2QWEA5VS6B2SQCMPOPZU6D3B2DHCDSVD7K6Q4QHFX44A, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:05.650766284 (565h33m5.521849329s) [upload=495851]: Next returned nil segment Dec 29 05:02:05.650794408 (565h33m5.521877453s) [upload=495851]: Waiting for error group managing segments... Dec 29 05:02:05.930115341 (565h33m5.801198376s) [noise=true , address=145.79.227.4:4185 , piece_id=YIWQKK6UWQYLUEZYM4RWYREFHVYGTVBTGNKT3OHNAGIZ63V7ZDOA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.936825428 (565h33m5.807908463s) [noise=true , address=145.79.227.17:4972 , piece_id=FNQJGEWU4WJU6PQLOGESRX62CC6GW3AZ774EPRLMRIAG4YHP6HFA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.951504720 (565h33m5.822587775s) [noise=true , address=145.79.227.51:4411 , piece_id=N2NMHY2ARKS6VF36ESHBVZZZ5WQOFZL7B54GZOYTI3XO4JSZAYNA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.952360490 (565h33m5.823443535s) [noise=true , address=145.79.227.41:4085 , piece_id=3XZ3M44UQWUJQVNFQ7U3UUMKCWP5F3LP45VC5RDCLR7POBL7D55A, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.953443091 (565h33m5.824526126s) [noise=true , address=145.79.227.38:4977 , piece_id=L6B3W4NBKZD2W7GGYATATRXF74F2G7EYR6U33RLRBKTYGVGEZEBQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.954147141 (565h33m5.825230196s) [noise=true , address=145.79.227.16:4912 , piece_id=OGXZZQ4S2QWEA5VS6B2SQCMPOPZU6D3B2DHCDSVD7K6Q4QHFX44A, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.956890145 (565h33m5.82797319s) [noise=true , address=145.79.227.15:4873 , piece_id=VN3CYY7J7FXQMZFWP356DBEJONINKOXAIAUMCPLFWDT3JOFXVN2Q, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.967715939 (565h33m5.838798974s) [noise=true , address=145.79.227.11:4730 , piece_id=6RFVK2UTGIZYYPWNYIGJ32HIDA5FJBWHSTUWCJOM2B6ZJ4TCIVJQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.973899664 (565h33m5.844982709s) [noise=true , address=145.79.227.32:4835 , piece_id=EWTEJ6COP33NYV3KJ6G33VHE74NWUUVW5CUNGPPKSW2WOT2NLUNQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.997147833 (565h33m5.868230878s) [noise=true , address=145.79.227.37:4566 , piece_id=6MLSDZPOLCKKE2L554QYOIXPFIBQ3MVAEYEJ3NP3XE4UKEVDM3UQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.237297548 (565h33m6.108380603s) [noise=true , address=45.91.116.11:4117 , piece_id=POSJ3MZUV2LHSZIDWTRS74H7TGEFLXINT3MECMLJ63IWDRZDRNBA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.249321813 (565h33m6.120404849s) [noise=true , address=45.91.116.5:4297 , piece_id=KSHKJABPELNNJM3PI452PEHTLD4NP4HFXNJOKAQKSUFLEVZGMFJQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.333711816 (565h33m6.204794861s) [noise=true , address=77.90.31.21:4878 , piece_id=IHR4NYH2ITJOOMBLMTV3LZKUH4FYAREZVJV2ETZR3VIZCQK455UA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.333738558 (565h33m6.204821603s) [seg_pos={0 0}, upload=495851]: Segment reached optimal threshold of 13 pieces. Dec 29 05:02:06.334180178 (565h33m6.205263223s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334189266 (565h33m6.205272311s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334409936 (565h33m6.205492981s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334417921 (565h33m6.205500976s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334913925 (565h33m6.20599697s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334922833 (565h33m6.206005888s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334947569 (565h33m6.206030604s) [seg_pos={0 0}, upload=495851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:02:06.335035306 (565h33m6.206118352s) [upload=495851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:02:06.335042801 (565h33m6.206125836s) [upload=495851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:02:06.631244788 (565h33m6.502327823s) [upload=495851]: Done uploading segments... Dec 29 05:02:06.631265338 (565h33m6.502348383s) [upload=495851]: Upload finished. err: <nil> Duration: 6.183544819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X...X] duration: 30.003015713s trace: https://jaeger.dev.storj.io/trace/2d957f66117cb224 FAIL (2025-12-29 05:02:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 05:02:03.695864504 (565h33m3.56694754s) [upload=495851]: Starting upload Dec 29 05:02:03.695969835 (565h33m3.56705288s) [upload=495851]: Done starting upload Dec 29 05:02:03.696142474 (565h33m3.567225519s) [upload=495851]: Uploading segments... Dec 29 05:02:03.696247293 (565h33m3.567330348s) [upload=495851]: Got next segment. Inline: false Dec 29 05:02:03.696294584 (565h33m3.567377629s) [upload=495851]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 05:02:03.696302739 (565h33m3.567385784s) [upload=495851]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 05:02:03.951067754 (565h33m3.822150799s) [seg_pos={0 0}, upload=495851]: Begin upload segment... Dec 29 05:02:03.951150241 (565h33m3.822233286s) [seg_pos={0 0}, upload=495851]: Done begin upload segment. Dec 29 05:02:03.951215325 (565h33m3.82229837s) [noise=true , address=45.91.116.11:4117 , piece_id=POSJ3MZUV2LHSZIDWTRS74H7TGEFLXINT3MECMLJ63IWDRZDRNBA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.951224143 (565h33m3.822307178s) [noise=true , address=45.91.116.5:4297 , piece_id=KSHKJABPELNNJM3PI452PEHTLD4NP4HFXNJOKAQKSUFLEVZGMFJQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.951739273 (565h33m3.822822308s) [noise=true , address=77.90.31.21:4878 , piece_id=IHR4NYH2ITJOOMBLMTV3LZKUH4FYAREZVJV2ETZR3VIZCQK455UA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952038814 (565h33m3.823121859s) [noise=true , address=145.79.227.11:4730 , piece_id=6RFVK2UTGIZYYPWNYIGJ32HIDA5FJBWHSTUWCJOM2B6ZJ4TCIVJQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952399821 (565h33m3.823482866s) [noise=true , address=145.79.227.4:4185 , piece_id=YIWQKK6UWQYLUEZYM4RWYREFHVYGTVBTGNKT3OHNAGIZ63V7ZDOA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.952817827 (565h33m3.823900862s) [noise=true , address=145.79.227.17:4972 , piece_id=FNQJGEWU4WJU6PQLOGESRX62CC6GW3AZ774EPRLMRIAG4YHP6HFA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.953148727 (565h33m3.824231762s) [noise=true , address=145.79.227.41:4085 , piece_id=3XZ3M44UQWUJQVNFQ7U3UUMKCWP5F3LP45VC5RDCLR7POBL7D55A, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.954432772 (565h33m3.825515817s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955051461 (565h33m3.826134506s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955496788 (565h33m3.826579833s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.955940753 (565h33m3.827023798s) [noise=true , address=145.79.227.32:4835 , piece_id=EWTEJ6COP33NYV3KJ6G33VHE74NWUUVW5CUNGPPKSW2WOT2NLUNQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.956460483 (565h33m3.827543528s) [noise=true , address=145.79.227.38:4977 , piece_id=L6B3W4NBKZD2W7GGYATATRXF74F2G7EYR6U33RLRBKTYGVGEZEBQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.956931761 (565h33m3.828014806s) [noise=true , address=145.79.227.51:4411 , piece_id=N2NMHY2ARKS6VF36ESHBVZZZ5WQOFZL7B54GZOYTI3XO4JSZAYNA, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.957418147 (565h33m3.828501192s) [noise=true , address=145.79.227.15:4873 , piece_id=VN3CYY7J7FXQMZFWP356DBEJONINKOXAIAUMCPLFWDT3JOFXVN2Q, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.957975589 (565h33m3.829058634s) [noise=true , address=145.79.227.37:4566 , piece_id=6MLSDZPOLCKKE2L554QYOIXPFIBQ3MVAEYEJ3NP3XE4UKEVDM3UQ, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:03.958731778 (565h33m3.829814843s) [noise=true , address=145.79.227.16:4912 , piece_id=OGXZZQ4S2QWEA5VS6B2SQCMPOPZU6D3B2DHCDSVD7K6Q4QHFX44A, seg_pos={0 0}, upload=495851]: Uploading piece... Dec 29 05:02:05.650766284 (565h33m5.521849329s) [upload=495851]: Next returned nil segment Dec 29 05:02:05.650794408 (565h33m5.521877453s) [upload=495851]: Waiting for error group managing segments... Dec 29 05:02:05.930115341 (565h33m5.801198376s) [noise=true , address=145.79.227.4:4185 , piece_id=YIWQKK6UWQYLUEZYM4RWYREFHVYGTVBTGNKT3OHNAGIZ63V7ZDOA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.936825428 (565h33m5.807908463s) [noise=true , address=145.79.227.17:4972 , piece_id=FNQJGEWU4WJU6PQLOGESRX62CC6GW3AZ774EPRLMRIAG4YHP6HFA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.951504720 (565h33m5.822587775s) [noise=true , address=145.79.227.51:4411 , piece_id=N2NMHY2ARKS6VF36ESHBVZZZ5WQOFZL7B54GZOYTI3XO4JSZAYNA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.952360490 (565h33m5.823443535s) [noise=true , address=145.79.227.41:4085 , piece_id=3XZ3M44UQWUJQVNFQ7U3UUMKCWP5F3LP45VC5RDCLR7POBL7D55A, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.953443091 (565h33m5.824526126s) [noise=true , address=145.79.227.38:4977 , piece_id=L6B3W4NBKZD2W7GGYATATRXF74F2G7EYR6U33RLRBKTYGVGEZEBQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.954147141 (565h33m5.825230196s) [noise=true , address=145.79.227.16:4912 , piece_id=OGXZZQ4S2QWEA5VS6B2SQCMPOPZU6D3B2DHCDSVD7K6Q4QHFX44A, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.956890145 (565h33m5.82797319s) [noise=true , address=145.79.227.15:4873 , piece_id=VN3CYY7J7FXQMZFWP356DBEJONINKOXAIAUMCPLFWDT3JOFXVN2Q, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.967715939 (565h33m5.838798974s) [noise=true , address=145.79.227.11:4730 , piece_id=6RFVK2UTGIZYYPWNYIGJ32HIDA5FJBWHSTUWCJOM2B6ZJ4TCIVJQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.973899664 (565h33m5.844982709s) [noise=true , address=145.79.227.32:4835 , piece_id=EWTEJ6COP33NYV3KJ6G33VHE74NWUUVW5CUNGPPKSW2WOT2NLUNQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:05.997147833 (565h33m5.868230878s) [noise=true , address=145.79.227.37:4566 , piece_id=6MLSDZPOLCKKE2L554QYOIXPFIBQ3MVAEYEJ3NP3XE4UKEVDM3UQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.237297548 (565h33m6.108380603s) [noise=true , address=45.91.116.11:4117 , piece_id=POSJ3MZUV2LHSZIDWTRS74H7TGEFLXINT3MECMLJ63IWDRZDRNBA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.249321813 (565h33m6.120404849s) [noise=true , address=45.91.116.5:4297 , piece_id=KSHKJABPELNNJM3PI452PEHTLD4NP4HFXNJOKAQKSUFLEVZGMFJQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.333711816 (565h33m6.204794861s) [noise=true , address=77.90.31.21:4878 , piece_id=IHR4NYH2ITJOOMBLMTV3LZKUH4FYAREZVJV2ETZR3VIZCQK455UA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: <nil> Dec 29 05:02:06.333738558 (565h33m6.204821603s) [seg_pos={0 0}, upload=495851]: Segment reached optimal threshold of 13 pieces. Dec 29 05:02:06.334180178 (565h33m6.205263223s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334189266 (565h33m6.205272311s) [noise=true , address=45.91.116.6:4975 , piece_id=FXBFK2UPLU7H25V55IUOEUANJKCQXM4HPIGYLSJHRMGLXGO6ISPQ, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334409936 (565h33m6.205492981s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334417921 (565h33m6.205500976s) [noise=true , address=145.79.227.33:4870 , piece_id=6ZVCLBP72PWL7MIR6PWPW7H5LMQW5YD2K5JVFV26PZRI54IFAS4Q, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334913925 (565h33m6.20599697s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 29 05:02:06.334922833 (565h33m6.206005888s) [noise=true , address=45.91.116.4:4937 , piece_id=A6M5CTMA2HEQ57266ELBRXLJIU7W7CG7L73PAVRQMCLCO46BOYNA, seg_pos={0 0}, upload=495851]: Long tail cancellation due to optimal threshold Dec 29 05:02:06.334947569 (565h33m6.206030604s) [seg_pos={0 0}, upload=495851]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 05:02:06.335035306 (565h33m6.206118352s) [upload=495851]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 05:02:06.335042801 (565h33m6.206125836s) [upload=495851]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 05:02:06.631244788 (565h33m6.502327823s) [upload=495851]: Done uploading segments... Dec 29 05:02:06.631265338 (565h33m6.502348383s) [upload=495851]: Upload finished. err: <nil> Duration: 30.00304027s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....XX.......X.....X..XX......X.X.....X.....X........X..X...] duration: 5.001659023s trace: https://jaeger.dev.storj.io/trace/273e007e7c4cd805 FAIL (2025-12-29 05:02:25 UTC) orbiter warning: context deadline exceeded Duration: 5.001684511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....XX.......X.....X..XX......X.X.....X.....X........X..X..] duration: 4.960945077s trace: https://jaeger.dev.storj.io/trace/273e007e7c4cd805 FAIL (2025-12-29 05:02:06 UTC) orbiter warning: context deadline exceeded Duration: 4.96096835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X...] duration: 5.838053707s trace: https://jaeger.dev.storj.io/trace/17e8560bc77aa232 FAIL (2025-12-29 05:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ) context deadline exceeded ======LOGS====== Dec 29 04:58:50.188956006 (565h29m50.060039061s) [upload=495806]: Starting upload Dec 29 04:58:50.189077507 (565h29m50.060160542s) [upload=495806]: Done starting upload Dec 29 04:58:50.189168631 (565h29m50.060251666s) [upload=495806]: Uploading segments... Dec 29 04:58:50.189346770 (565h29m50.060429805s) [upload=495806]: Got next segment. Inline: false Dec 29 04:58:50.189400543 (565h29m50.060483588s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:58:50.189406374 (565h29m50.060489409s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:58:50.437653067 (565h29m50.308736102s) [seg_pos={0 0}, upload=495806]: Begin upload segment... Dec 29 04:58:50.437707931 (565h29m50.308790976s) [seg_pos={0 0}, upload=495806]: Done begin upload segment. Dec 29 04:58:50.437766513 (565h29m50.308849548s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.437773426 (565h29m50.308856461s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.438238282 (565h29m50.309321327s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439201064 (565h29m50.310284109s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439403380 (565h29m50.310486415s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439775238 (565h29m50.310858303s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.440603215 (565h29m50.31168629s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.441633907 (565h29m50.312716942s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.442767236 (565h29m50.313850271s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443359083 (565h29m50.314442118s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443923407 (565h29m50.315006442s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444276870 (565h29m50.315359905s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444781541 (565h29m50.315864576s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.445479420 (565h29m50.316562455s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.446648225 (565h29m50.31773126s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.447045262 (565h29m50.318128297s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:52.143872213 (565h29m52.014955258s) [upload=495806]: Next returned nil segment Dec 29 04:58:52.143885438 (565h29m52.014968473s) [upload=495806]: Waiting for error group managing segments... Dec 29 04:58:52.433448194 (565h29m52.304531229s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435499190 (565h29m52.306582225s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435769844 (565h29m52.306852889s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.446095897 (565h29m52.317178932s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.453872595 (565h29m52.32495563s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.456444412 (565h29m52.327527447s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.462220150 (565h29m52.333303185s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.463303844 (565h29m52.334386879s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.471472629 (565h29m52.342555664s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.479942126 (565h29m52.351025161s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.679789237 (565h29m52.550872272s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.716435839 (565h29m52.587518884s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772311134 (565h29m52.643394169s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772337705 (565h29m52.64342074s) [seg_pos={0 0}, upload=495806]: Segment reached optimal threshold of 13 pieces. Dec 29 04:58:52.772761662 (565h29m52.643844697s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772793683 (565h29m52.643876718s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.772838780 (565h29m52.643921815s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772849680 (565h29m52.643932715s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773068527 (565h29m52.644151562s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.773076692 (565h29m52.644159737s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773101339 (565h29m52.644184374s) [seg_pos={0 0}, upload=495806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:58:52.773203965 (565h29m52.64428701s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:58:52.773211439 (565h29m52.644294494s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:58:53.064101392 (565h29m52.935184447s) [upload=495806]: Done uploading segments... Dec 29 04:58:53.064127501 (565h29m52.935210546s) [upload=495806]: Upload finished. err: <nil> Duration: 5.83808183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......XX.......X.....X..XX......X.X.....X.....X........X..X.] duration: 3.628001052s trace: https://jaeger.dev.storj.io/trace/273e007e7c4cd805 FAIL (2025-12-29 05:01:17 UTC) orbiter warning: context deadline exceeded Duration: 3.628027712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X..] duration: 5.98592629s trace: https://jaeger.dev.storj.io/trace/17e8560bc77aa232 FAIL (2025-12-29 05:00:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ) context deadline exceeded ======LOGS====== Dec 29 04:58:50.188956006 (565h29m50.060039061s) [upload=495806]: Starting upload Dec 29 04:58:50.189077507 (565h29m50.060160542s) [upload=495806]: Done starting upload Dec 29 04:58:50.189168631 (565h29m50.060251666s) [upload=495806]: Uploading segments... Dec 29 04:58:50.189346770 (565h29m50.060429805s) [upload=495806]: Got next segment. Inline: false Dec 29 04:58:50.189400543 (565h29m50.060483588s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:58:50.189406374 (565h29m50.060489409s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:58:50.437653067 (565h29m50.308736102s) [seg_pos={0 0}, upload=495806]: Begin upload segment... Dec 29 04:58:50.437707931 (565h29m50.308790976s) [seg_pos={0 0}, upload=495806]: Done begin upload segment. Dec 29 04:58:50.437766513 (565h29m50.308849548s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.437773426 (565h29m50.308856461s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.438238282 (565h29m50.309321327s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439201064 (565h29m50.310284109s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439403380 (565h29m50.310486415s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439775238 (565h29m50.310858303s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.440603215 (565h29m50.31168629s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.441633907 (565h29m50.312716942s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.442767236 (565h29m50.313850271s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443359083 (565h29m50.314442118s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443923407 (565h29m50.315006442s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444276870 (565h29m50.315359905s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444781541 (565h29m50.315864576s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.445479420 (565h29m50.316562455s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.446648225 (565h29m50.31773126s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.447045262 (565h29m50.318128297s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:52.143872213 (565h29m52.014955258s) [upload=495806]: Next returned nil segment Dec 29 04:58:52.143885438 (565h29m52.014968473s) [upload=495806]: Waiting for error group managing segments... Dec 29 04:58:52.433448194 (565h29m52.304531229s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435499190 (565h29m52.306582225s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435769844 (565h29m52.306852889s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.446095897 (565h29m52.317178932s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.453872595 (565h29m52.32495563s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.456444412 (565h29m52.327527447s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.462220150 (565h29m52.333303185s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.463303844 (565h29m52.334386879s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.471472629 (565h29m52.342555664s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.479942126 (565h29m52.351025161s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.679789237 (565h29m52.550872272s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.716435839 (565h29m52.587518884s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772311134 (565h29m52.643394169s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772337705 (565h29m52.64342074s) [seg_pos={0 0}, upload=495806]: Segment reached optimal threshold of 13 pieces. Dec 29 04:58:52.772761662 (565h29m52.643844697s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772793683 (565h29m52.643876718s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.772838780 (565h29m52.643921815s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772849680 (565h29m52.643932715s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773068527 (565h29m52.644151562s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.773076692 (565h29m52.644159737s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773101339 (565h29m52.644184374s) [seg_pos={0 0}, upload=495806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:58:52.773203965 (565h29m52.64428701s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:58:52.773211439 (565h29m52.644294494s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:58:53.064101392 (565h29m52.935184447s) [upload=495806]: Done uploading segments... Dec 29 04:58:53.064127501 (565h29m52.935210546s) [upload=495806]: Upload finished. err: <nil> Duration: 5.985950164s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.......XX.......X.....X..XX......X.X.....X.....X........X..X] duration: 5.001147851s trace: https://jaeger.dev.storj.io/trace/273e007e7c4cd805 FAIL (2025-12-29 05:00:11 UTC) orbiter warning: context deadline exceeded Duration: 5.00117345s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [........XX.......X.....X..XX......X.X.....X.....X........X..] duration: 3.598154466s trace: https://jaeger.dev.storj.io/trace/43e64e1035c8b7cc FAIL (2025-12-29 04:59:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59817717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X.] duration: 6.01129136s trace: https://jaeger.dev.storj.io/trace/17e8560bc77aa232 FAIL (2025-12-29 04:59:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ) context deadline exceeded ======LOGS====== Dec 29 04:58:50.188956006 (565h29m50.060039061s) [upload=495806]: Starting upload Dec 29 04:58:50.189077507 (565h29m50.060160542s) [upload=495806]: Done starting upload Dec 29 04:58:50.189168631 (565h29m50.060251666s) [upload=495806]: Uploading segments... Dec 29 04:58:50.189346770 (565h29m50.060429805s) [upload=495806]: Got next segment. Inline: false Dec 29 04:58:50.189400543 (565h29m50.060483588s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:58:50.189406374 (565h29m50.060489409s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:58:50.437653067 (565h29m50.308736102s) [seg_pos={0 0}, upload=495806]: Begin upload segment... Dec 29 04:58:50.437707931 (565h29m50.308790976s) [seg_pos={0 0}, upload=495806]: Done begin upload segment. Dec 29 04:58:50.437766513 (565h29m50.308849548s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.437773426 (565h29m50.308856461s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.438238282 (565h29m50.309321327s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439201064 (565h29m50.310284109s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439403380 (565h29m50.310486415s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439775238 (565h29m50.310858303s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.440603215 (565h29m50.31168629s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.441633907 (565h29m50.312716942s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.442767236 (565h29m50.313850271s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443359083 (565h29m50.314442118s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443923407 (565h29m50.315006442s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444276870 (565h29m50.315359905s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444781541 (565h29m50.315864576s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.445479420 (565h29m50.316562455s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.446648225 (565h29m50.31773126s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.447045262 (565h29m50.318128297s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:52.143872213 (565h29m52.014955258s) [upload=495806]: Next returned nil segment Dec 29 04:58:52.143885438 (565h29m52.014968473s) [upload=495806]: Waiting for error group managing segments... Dec 29 04:58:52.433448194 (565h29m52.304531229s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435499190 (565h29m52.306582225s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435769844 (565h29m52.306852889s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.446095897 (565h29m52.317178932s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.453872595 (565h29m52.32495563s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.456444412 (565h29m52.327527447s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.462220150 (565h29m52.333303185s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.463303844 (565h29m52.334386879s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.471472629 (565h29m52.342555664s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.479942126 (565h29m52.351025161s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.679789237 (565h29m52.550872272s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.716435839 (565h29m52.587518884s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772311134 (565h29m52.643394169s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772337705 (565h29m52.64342074s) [seg_pos={0 0}, upload=495806]: Segment reached optimal threshold of 13 pieces. Dec 29 04:58:52.772761662 (565h29m52.643844697s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772793683 (565h29m52.643876718s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.772838780 (565h29m52.643921815s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772849680 (565h29m52.643932715s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773068527 (565h29m52.644151562s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.773076692 (565h29m52.644159737s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773101339 (565h29m52.644184374s) [seg_pos={0 0}, upload=495806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:58:52.773203965 (565h29m52.64428701s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:58:52.773211439 (565h29m52.644294494s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:58:53.064101392 (565h29m52.935184447s) [upload=495806]: Done uploading segments... Dec 29 04:58:53.064127501 (565h29m52.935210546s) [upload=495806]: Upload finished. err: <nil> Duration: 6.011306649s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX......X] duration: 30.003226923s trace: https://jaeger.dev.storj.io/trace/17e8560bc77aa232 FAIL (2025-12-29 04:59:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ) context deadline exceeded ======LOGS====== Dec 29 04:58:50.188956006 (565h29m50.060039061s) [upload=495806]: Starting upload Dec 29 04:58:50.189077507 (565h29m50.060160542s) [upload=495806]: Done starting upload Dec 29 04:58:50.189168631 (565h29m50.060251666s) [upload=495806]: Uploading segments... Dec 29 04:58:50.189346770 (565h29m50.060429805s) [upload=495806]: Got next segment. Inline: false Dec 29 04:58:50.189400543 (565h29m50.060483588s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:58:50.189406374 (565h29m50.060489409s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:58:50.437653067 (565h29m50.308736102s) [seg_pos={0 0}, upload=495806]: Begin upload segment... Dec 29 04:58:50.437707931 (565h29m50.308790976s) [seg_pos={0 0}, upload=495806]: Done begin upload segment. Dec 29 04:58:50.437766513 (565h29m50.308849548s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.437773426 (565h29m50.308856461s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.438238282 (565h29m50.309321327s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439201064 (565h29m50.310284109s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439403380 (565h29m50.310486415s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.439775238 (565h29m50.310858303s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.440603215 (565h29m50.31168629s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.441633907 (565h29m50.312716942s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.442767236 (565h29m50.313850271s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443359083 (565h29m50.314442118s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.443923407 (565h29m50.315006442s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444276870 (565h29m50.315359905s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.444781541 (565h29m50.315864576s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.445479420 (565h29m50.316562455s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.446648225 (565h29m50.31773126s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:50.447045262 (565h29m50.318128297s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Uploading piece... Dec 29 04:58:52.143872213 (565h29m52.014955258s) [upload=495806]: Next returned nil segment Dec 29 04:58:52.143885438 (565h29m52.014968473s) [upload=495806]: Waiting for error group managing segments... Dec 29 04:58:52.433448194 (565h29m52.304531229s) [noise=true , address=145.79.227.14:4839 , piece_id=MOLSEE2IZK3B7NONB4CFX3QHAZZ4WDHJKLYLNPIAI6ELWXJTD7HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435499190 (565h29m52.306582225s) [noise=true , address=145.79.227.7:4294 , piece_id=P7GCWX6EFSFSYTAL3I2CVX34VQC64QUFGCCBRUXM5WCN3GGXYFSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.435769844 (565h29m52.306852889s) [noise=true , address=145.79.227.15:4874 , piece_id=ZXC3H4DXWJ7VZLP53QTDSEXAZCPBNXNKDEJDPFL4Q74ELXHLD6ZA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.446095897 (565h29m52.317178932s) [noise=true , address=145.79.227.38:4979 , piece_id=OSQ4RUPZV3ZMFBX74AEUVUM667YYHCRBQ422BMAYCRUATHPR76HA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.453872595 (565h29m52.32495563s) [noise=true , address=145.79.227.17:4972 , piece_id=QYL3Y7NRRB73N2XD5ASUPQ3UOZKOMFBH265CF7LY5QBYS6RPI2OQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.456444412 (565h29m52.327527447s) [noise=true , address=145.79.227.42:4122 , piece_id=3MNDTFIESTQQRHKSWE72WQVRMLXLTYPNL5XYPUO7WEKF2Z5BRNKQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.462220150 (565h29m52.333303185s) [noise=true , address=145.79.227.46:4602 , piece_id=UIBVETDCFUTE3AXGUXPGU53RKDCJT3M2GPYK55ZOM2TRZ5VAQMKA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.463303844 (565h29m52.334386879s) [noise=true , address=145.79.227.50:4375 , piece_id=6M33HEXPPHJ2VG6F47KZ2GOYOVPLKWPR7X26DHZCQNQU7BWOVLUQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.471472629 (565h29m52.342555664s) [noise=true , address=145.79.227.6:4258 , piece_id=SNB2D5RGI6YGEUBAOMHBN45DROHWMKIASVQZGPAK4GLFNXHG6BLQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.479942126 (565h29m52.351025161s) [noise=true , address=145.79.227.4:4187 , piece_id=VLMKHTET2P7SKT6V7HQO4TQRXV7XSGDOODCLPJY4YU74ALWXRDSQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.679789237 (565h29m52.550872272s) [noise=true , address=45.91.116.14:4189 , piece_id=YKVNAB2CSQZ6LLIO2IPNZ4KNKAJA636CY5W6GET5ZM66FR5FRTLA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.716435839 (565h29m52.587518884s) [noise=true , address=45.91.116.4:4940 , piece_id=FZIFTE7O4UYZHDWTCMYJ37GRT7HUENIMFSWG6PBFQI4L4MZFZJ4A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772311134 (565h29m52.643394169s) [noise=true , address=45.91.116.8:4048 , piece_id=F7YYNGR4FDTOQEVFKPQR5ZEDT72E3RLOZ4JQAO5KNGVN6YT4ITDQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: <nil> Dec 29 04:58:52.772337705 (565h29m52.64342074s) [seg_pos={0 0}, upload=495806]: Segment reached optimal threshold of 13 pieces. Dec 29 04:58:52.772761662 (565h29m52.643844697s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772793683 (565h29m52.643876718s) [noise=true , address=77.90.31.11:4553 , piece_id=WPOZ6FZZ3NW2EHJMC65WNFGEGKYIDR6I4DCMGR3QL3NZWUDKU2SA, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.772838780 (565h29m52.643921815s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.772849680 (565h29m52.643932715s) [noise=true , address=45.91.116.12:4225 , piece_id=SIC7HQ3ADSHIHYFVKFJHYGW3X55XAK5IGXEGJ6SMCWMANYHTAX5A, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773068527 (565h29m52.644151562s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 29 04:58:52.773076692 (565h29m52.644159737s) [noise=true , address=45.91.116.9:4261 , piece_id=SL2L3IDRDKKJUJWEA5NXRQ66PDPJHYJDDTUJAGHW5P552LGAN4VQ, seg_pos={0 0}, upload=495806]: Long tail cancellation due to optimal threshold Dec 29 04:58:52.773101339 (565h29m52.644184374s) [seg_pos={0 0}, upload=495806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:58:52.773203965 (565h29m52.64428701s) [upload=495806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:58:52.773211439 (565h29m52.644294494s) [upload=495806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:58:53.064101392 (565h29m52.935184447s) [upload=495806]: Done uploading segments... Dec 29 04:58:53.064127501 (565h29m52.935210546s) [upload=495806]: Upload finished. err: <nil> Duration: 30.003248664s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.........XX.......X.....X..XX......X.X.....X.....X........X.] duration: 3.388561595s trace: https://jaeger.dev.storj.io/trace/43e64e1035c8b7cc FAIL (2025-12-29 04:58:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.388589266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX......] duration: 20.095658894s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:58:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 20.095684211s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.........XX.......X.....X..XX......X.X.....X.....X........X] duration: 5.001750495s trace: https://jaeger.dev.storj.io/trace/43e64e1035c8b7cc FAIL (2025-12-29 04:58:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001775302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X.........XX.......X.....X..XX......X.X.....X.....X........] duration: 3.681404027s trace: https://jaeger.dev.storj.io/trace/1dc4e09bac304a10 FAIL (2025-12-29 04:58:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.681430448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX.....] duration: 10.680861029s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:57:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 10.68088737s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..X.........XX.......X.....X..XX......X.X.....X.....X.......] duration: 3.484616147s trace: https://jaeger.dev.storj.io/trace/1dc4e09bac304a10 FAIL (2025-12-29 04:57:25 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.484692253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX....] duration: 5.791776624s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 5.791804236s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X.........XX.......X.....X..XX......X.X.....X.....X......] duration: 3.587294039s trace: https://jaeger.dev.storj.io/trace/1dc4e09bac304a10 FAIL (2025-12-29 04:56:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.587324999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX...] duration: 5.972629465s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 5.972655614s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 15.492025558s trace: https://jaeger.dev.storj.io/trace/72beea211c54dceb FAIL (2025-12-29 04:55:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:05:05.301177067 (564h36m5.172260112s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301468672 (564h36m5.172551717s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301878142 (564h36m5.172961187s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301892169 (564h36m5.172975214s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302216927 (564h36m5.173299972s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302327137 (564h36m5.173410182s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302684628 (564h36m5.173767673s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302932379 (564h36m5.174015424s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303033903 (564h36m5.174116948s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303385993 (564h36m5.174469038s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303457709 (564h36m5.174540754s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303825861 (564h36m5.174908906s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303866187 (564h36m5.174949233s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304215443 (564h36m5.175298488s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304302198 (564h36m5.175385243s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304739651 (564h36m5.175822696s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:06.612294683 (564h36m6.483377728s) [upload=495011]: Next returned nil segment Dec 29 04:05:06.612312939 (564h36m6.483395984s) [upload=495011]: Waiting for error group managing segments... Dec 29 04:05:07.137646212 (564h36m7.008729257s) [noise=true , address=145.79.227.6:4266 , piece_id=VARLMNUO5V4T5KFYD6BQ3XHENCVVPQNRVYTWNV3NCGVHGTRHK75Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.153061636 (564h36m7.024144681s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.174839534 (564h36m7.045922579s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.196190780 (564h36m7.067273825s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.439229803 (564h36m7.310312839s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.458345231 (564h36m7.329428266s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.461742912 (564h36m7.332825957s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.501492300 (564h36m7.372575335s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.520950161 (564h36m7.392033196s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.900960668 (564h36m7.772043713s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.070127046 (564h36m7.941210081s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.116327979 (564h36m7.987411015s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.117842052 (564h36m7.988925097s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145319343 (564h36m8.016402378s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145349340 (564h36m8.016432386s) [seg_pos={0 0}, upload=495011]: Segment reached optimal threshold of 14 pieces. Dec 29 04:05:08.145520887 (564h36m8.016603932s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.145527079 (564h36m8.016610124s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.145961806 (564h36m8.017044851s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection Dec 29 04:05:08.145971706 (564h36m8.017054741s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146249024 (564h36m8.017332069s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146259193 (564h36m8.017342248s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146601144 (564h36m8.017684189s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146612105 (564h36m8.01769515s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146660708 (564h36m8.017743753s) [seg_pos={0 0}, upload=495011]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 04:05:08.146791666 (564h36m8.017874712s) [upload=495011]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:05:08.146799432 (564h36m8.017882477s) [upload=495011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:05:08.432716311 (564h36m8.303799356s) [upload=495011]: Done uploading segments... Dec 29 04:05:08.432736209 (564h36m8.303819254s) [upload=495011]: Upload finished. err: <nil> Duration: 15.492053061s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X...X.........XX.......X.....X..XX......X.X.....X.....X.....] duration: 4.723141263s trace: https://jaeger.dev.storj.io/trace/1dc4e09bac304a10 FAIL (2025-12-29 04:55:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.723165238s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 26.648952856s trace: https://jaeger.dev.storj.io/trace/72beea211c54dceb FAIL (2025-12-29 04:55:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:05:05.301177067 (564h36m5.172260112s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301468672 (564h36m5.172551717s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301878142 (564h36m5.172961187s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301892169 (564h36m5.172975214s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302216927 (564h36m5.173299972s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302327137 (564h36m5.173410182s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302684628 (564h36m5.173767673s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302932379 (564h36m5.174015424s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303033903 (564h36m5.174116948s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303385993 (564h36m5.174469038s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303457709 (564h36m5.174540754s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303825861 (564h36m5.174908906s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303866187 (564h36m5.174949233s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304215443 (564h36m5.175298488s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304302198 (564h36m5.175385243s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304739651 (564h36m5.175822696s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:06.612294683 (564h36m6.483377728s) [upload=495011]: Next returned nil segment Dec 29 04:05:06.612312939 (564h36m6.483395984s) [upload=495011]: Waiting for error group managing segments... Dec 29 04:05:07.137646212 (564h36m7.008729257s) [noise=true , address=145.79.227.6:4266 , piece_id=VARLMNUO5V4T5KFYD6BQ3XHENCVVPQNRVYTWNV3NCGVHGTRHK75Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.153061636 (564h36m7.024144681s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.174839534 (564h36m7.045922579s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.196190780 (564h36m7.067273825s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.439229803 (564h36m7.310312839s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.458345231 (564h36m7.329428266s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.461742912 (564h36m7.332825957s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.501492300 (564h36m7.372575335s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.520950161 (564h36m7.392033196s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.900960668 (564h36m7.772043713s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.070127046 (564h36m7.941210081s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.116327979 (564h36m7.987411015s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.117842052 (564h36m7.988925097s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145319343 (564h36m8.016402378s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145349340 (564h36m8.016432386s) [seg_pos={0 0}, upload=495011]: Segment reached optimal threshold of 14 pieces. Dec 29 04:05:08.145520887 (564h36m8.016603932s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.145527079 (564h36m8.016610124s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.145961806 (564h36m8.017044851s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection Dec 29 04:05:08.145971706 (564h36m8.017054741s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146249024 (564h36m8.017332069s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146259193 (564h36m8.017342248s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146601144 (564h36m8.017684189s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146612105 (564h36m8.01769515s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146660708 (564h36m8.017743753s) [seg_pos={0 0}, upload=495011]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 04:05:08.146791666 (564h36m8.017874712s) [upload=495011]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:05:08.146799432 (564h36m8.017882477s) [upload=495011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:05:08.432716311 (564h36m8.303799356s) [upload=495011]: Done uploading segments... Dec 29 04:05:08.432736209 (564h36m8.303819254s) [upload=495011]: Upload finished. err: <nil> Duration: 26.648979316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 41.725924069s trace: https://jaeger.dev.storj.io/trace/72beea211c54dceb FAIL (2025-12-29 04:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:05:05.301177067 (564h36m5.172260112s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301468672 (564h36m5.172551717s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301878142 (564h36m5.172961187s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301892169 (564h36m5.172975214s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302216927 (564h36m5.173299972s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302327137 (564h36m5.173410182s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302684628 (564h36m5.173767673s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302932379 (564h36m5.174015424s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303033903 (564h36m5.174116948s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303385993 (564h36m5.174469038s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303457709 (564h36m5.174540754s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303825861 (564h36m5.174908906s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303866187 (564h36m5.174949233s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304215443 (564h36m5.175298488s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304302198 (564h36m5.175385243s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304739651 (564h36m5.175822696s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:06.612294683 (564h36m6.483377728s) [upload=495011]: Next returned nil segment Dec 29 04:05:06.612312939 (564h36m6.483395984s) [upload=495011]: Waiting for error group managing segments... Dec 29 04:05:07.137646212 (564h36m7.008729257s) [noise=true , address=145.79.227.6:4266 , piece_id=VARLMNUO5V4T5KFYD6BQ3XHENCVVPQNRVYTWNV3NCGVHGTRHK75Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.153061636 (564h36m7.024144681s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.174839534 (564h36m7.045922579s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.196190780 (564h36m7.067273825s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.439229803 (564h36m7.310312839s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.458345231 (564h36m7.329428266s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.461742912 (564h36m7.332825957s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.501492300 (564h36m7.372575335s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.520950161 (564h36m7.392033196s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.900960668 (564h36m7.772043713s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.070127046 (564h36m7.941210081s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.116327979 (564h36m7.987411015s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.117842052 (564h36m7.988925097s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145319343 (564h36m8.016402378s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145349340 (564h36m8.016432386s) [seg_pos={0 0}, upload=495011]: Segment reached optimal threshold of 14 pieces. Dec 29 04:05:08.145520887 (564h36m8.016603932s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.145527079 (564h36m8.016610124s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.145961806 (564h36m8.017044851s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection Dec 29 04:05:08.145971706 (564h36m8.017054741s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146249024 (564h36m8.017332069s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146259193 (564h36m8.017342248s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146601144 (564h36m8.017684189s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146612105 (564h36m8.01769515s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146660708 (564h36m8.017743753s) [seg_pos={0 0}, upload=495011]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 04:05:08.146791666 (564h36m8.017874712s) [upload=495011]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:05:08.146799432 (564h36m8.017882477s) [upload=495011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:05:08.432716311 (564h36m8.303799356s) [upload=495011]: Done uploading segments... Dec 29 04:05:08.432736209 (564h36m8.303819254s) [upload=495011]: Upload finished. err: <nil> Duration: 41.726034801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX..] duration: 5.941968023s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:54:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 5.941994674s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X...X.........XX.......X.....X..XX......X.X.....X.....X....] duration: 3.481845007s trace: https://jaeger.dev.storj.io/trace/1dc4e09bac304a10 FAIL (2025-12-29 04:54:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481872209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X.X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX.] duration: 5.707996881s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:54:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 5.708024953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 43.494252496s trace: https://jaeger.dev.storj.io/trace/72beea211c54dceb FAIL (2025-12-29 04:53:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:05:05.301177067 (564h36m5.172260112s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301468672 (564h36m5.172551717s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301878142 (564h36m5.172961187s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.301892169 (564h36m5.172975214s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302216927 (564h36m5.173299972s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302327137 (564h36m5.173410182s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302684628 (564h36m5.173767673s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.302932379 (564h36m5.174015424s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303033903 (564h36m5.174116948s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303385993 (564h36m5.174469038s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303457709 (564h36m5.174540754s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303825861 (564h36m5.174908906s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.303866187 (564h36m5.174949233s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304215443 (564h36m5.175298488s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304302198 (564h36m5.175385243s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:05.304739651 (564h36m5.175822696s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Uploading piece... Dec 29 04:05:06.612294683 (564h36m6.483377728s) [upload=495011]: Next returned nil segment Dec 29 04:05:06.612312939 (564h36m6.483395984s) [upload=495011]: Waiting for error group managing segments... Dec 29 04:05:07.137646212 (564h36m7.008729257s) [noise=true , address=145.79.227.6:4266 , piece_id=VARLMNUO5V4T5KFYD6BQ3XHENCVVPQNRVYTWNV3NCGVHGTRHK75Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.153061636 (564h36m7.024144681s) [noise=true , address=145.79.227.46:4625 , piece_id=LHJG3OGMBKV7FIYCT2PHXRSGE2PBS2ZLBGMEPLCMHVXDXRPHQ4AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.174839534 (564h36m7.045922579s) [noise=true , address=145.79.227.2:4130 , piece_id=QVIKJQ7OOZJZYXWZ5UCJFQGXRJKXAWJ34IZ6PFKP46FLIBYI5UTQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.196190780 (564h36m7.067273825s) [noise=true , address=145.79.227.13:4831 , piece_id=AYEVGOIXOHA3U3ZUJB4DRLBXZRAQPHPM554QYOIF5CEKXT7SQ6HA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.439229803 (564h36m7.310312839s) [noise=true , address=45.91.116.9:4274 , piece_id=3WC4AZ6K3W45F6AYRPL7GXBEWMOAAVIQLVRKNBFK7ENMENZEPPJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.458345231 (564h36m7.329428266s) [noise=true , address=45.91.116.5:4309 , piece_id=SPLHWRKDB5XBYTMHA35AT3PW4MNVDPUJVTTKUGEY7Z5E4SSLAOJA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.461742912 (564h36m7.332825957s) [noise=true , address=77.90.31.17:4799 , piece_id=CKKO5SEZCKBQRLNFZVJKMLGCVDAFTPOICUSMVCDM7URTGFFKNYKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.501492300 (564h36m7.372575335s) [noise=true , address=45.91.116.10:4107 , piece_id=HQWXGMVKAU7LH5XBBJDM2VN3YBEZHFNYWH4NRRKBYGC7WYGBV6AQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.520950161 (564h36m7.392033196s) [noise=true , address=45.91.116.14:4204 , piece_id=6ZXLIAXBHNISDBIUAJV73SHVSFVBRI4N2BND67C6AVFUBJJHE45Q, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:07.900960668 (564h36m7.772043713s) [noise=true , address=193.149.164.143:4699 , piece_id=E7LTOGDKLAZHYNTU4KX4HBAYYT3MACHAAVH736JLJXRNN3OOJVKQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.070127046 (564h36m7.941210081s) [noise=true , address=77.90.31.15:4705 , piece_id=GJPXFW3E7UNYOOBQLG3TKZCWI6U7ORSAVFAU4MZ5RZBTN75LI6IQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.116327979 (564h36m7.987411015s) [noise=true , address=145.79.227.44:4201 , piece_id=H2QP7ULE7GIA5SDKJXEX5S7C5SVWP2474UP36JF4PXMPKTAMZVKA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.117842052 (564h36m7.988925097s) [noise=true , address=145.79.227.40:4081 , piece_id=WQKZITIAU7COGCM3XF7MM5JB64WB4GFMQKPUCBHNUY6IWJVWX4TA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145319343 (564h36m8.016402378s) [noise=true , address=45.91.116.8:4065 , piece_id=QGDKNF4XZYRPYLWZHUCP2IRJFR3E4A7ABSU7RUOK4NCEUBF2KKEQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: <nil> Dec 29 04:05:08.145349340 (564h36m8.016432386s) [seg_pos={0 0}, upload=495011]: Segment reached optimal threshold of 14 pieces. Dec 29 04:05:08.145520887 (564h36m8.016603932s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.145527079 (564h36m8.016610124s) [noise=true , address=145.79.227.30:4796 , piece_id=23D3V4P6NBGQMJBO7UWWUWD7ABA2URIZIMYRYRPTHOBQWQAHAGUA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.145961806 (564h36m8.017044851s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54220->64.49.13.169:4706: use of closed network connection Dec 29 04:05:08.145971706 (564h36m8.017054741s) [noise=true , address=64.49.13.169:4706 , piece_id=U7FALR7T6ZBLQWSZY55EMCTCVUQVOCE3FXTZKUF43MMUEGIGEBFQ, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146249024 (564h36m8.017332069s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146259193 (564h36m8.017342248s) [noise=true , address=145.79.227.33:4873 , piece_id=43LPXE4YYOYDHIJMJNZM4JZV56EXQVUDYIOQD37RMXFGBAWTN5IA, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146601144 (564h36m8.017684189s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU): protocol: expected piece hash; context canceled; EOF Dec 29 04:05:08.146612105 (564h36m8.01769515s) [noise=true , address=145.79.227.29:4947 , piece_id=5DJOP4RIA76SP32KTRL5GC6HWJS6YGQNYPBEWFXHYELLIYBA7U3A, seg_pos={0 0}, upload=495011]: Long tail cancellation due to optimal threshold Dec 29 04:05:08.146660708 (564h36m8.017743753s) [seg_pos={0 0}, upload=495011]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 04:05:08.146791666 (564h36m8.017874712s) [upload=495011]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:05:08.146799432 (564h36m8.017882477s) [upload=495011]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:05:08.432716311 (564h36m8.303799356s) [upload=495011]: Done uploading segments... Dec 29 04:05:08.432736209 (564h36m8.303819254s) [upload=495011]: Upload finished. err: <nil> Duration: 43.494281742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X.X.X.X.X...X.X..XX.X.....XX.......X.XX..XX..XXX........XX] duration: 30.003868705s trace: https://jaeger.dev.storj.io/trace/29c0060f8047887c FAIL (2025-12-29 04:53:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: 4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ) context deadline exceeded; piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ) context deadline exceeded ======LOGS====== Dec 29 04:53:23.068200176 (565h24m22.939283211s) [upload=495727]: Starting upload Dec 29 04:53:23.068260010 (565h24m22.939343055s) [upload=495727]: Done starting upload Dec 29 04:53:23.068288995 (565h24m22.93937203s) [upload=495727]: Uploading segments... Dec 29 04:53:23.068426446 (565h24m22.939509481s) [upload=495727]: Got next segment. Inline: false Dec 29 04:53:23.068452867 (565h24m22.939535902s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 04:53:23.068456694 (565h24m22.939539729s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 04:53:23.312266482 (565h24m23.183349517s) [seg_pos={0 0}, upload=495727]: Begin upload segment... Dec 29 04:53:23.312321015 (565h24m23.18340405s) [seg_pos={0 0}, upload=495727]: Done begin upload segment. Dec 29 04:53:23.312341976 (565h24m23.183425021s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312363387 (565h24m23.183446422s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312747648 (565h24m23.183830683s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.312974219 (565h24m23.184057254s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313031549 (565h24m23.184114594s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313294238 (565h24m23.184377273s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313433033 (565h24m23.184516068s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313626742 (565h24m23.184709777s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.313761759 (565h24m23.184844794s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314174023 (565h24m23.185257058s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314450029 (565h24m23.185533074s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.314775660 (565h24m23.185858695s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315074017 (565h24m23.186157052s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315393787 (565h24m23.186476832s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315686855 (565h24m23.18676989s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:23.315967990 (565h24m23.187051025s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Uploading piece... Dec 29 04:53:25.001362622 (565h24m24.872445688s) [upload=495727]: Next returned nil segment Dec 29 04:53:25.001388592 (565h24m24.872471627s) [upload=495727]: Waiting for error group managing segments... Dec 29 04:53:25.283455496 (565h24m25.154538531s) [noise=true , address=145.79.227.14:4840 , piece_id=567AC6V3T4DOXY6AUJVQOXOCEK5LI3TVB3CX5IQVIMK7UORKWVLA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.285944796 (565h24m25.157027831s) [noise=true , address=145.79.227.6:4257 , piece_id=NZG7GNEFAXSNLD4QQ6CQSXQBRNGLGI2MEVDX2O7ASUIKNXN6KFMA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.291789846 (565h24m25.162872911s) [noise=true , address=145.79.227.2:4114 , piece_id=PRSHK4YXAJVQ2NDEGBBYJAX3CKJ3WH234K7PPPJ6GEKWY2YRN63Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.297903857 (565h24m25.168986903s) [noise=true , address=145.79.227.37:4565 , piece_id=5EKVCR44WBKW2MSN5FOYUS2GYLAOB6QM2BLK4QYOLI3GIM3NQSOQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.311336456 (565h24m25.182419491s) [noise=true , address=145.79.227.19:4077 , piece_id=EABKSYBOBVT7DHL4Y4MXK3XMVGQHMGWPX4RGSCDCIKVJVIGXYNJA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.318152153 (565h24m25.189235188s) [noise=true , address=145.79.227.18:4007 , piece_id=VB6YC2XYHE3KRSGNPNOHOJHYY2VXWTJGJ7DGYXGU7PH4ZSYAGPHA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.319547630 (565h24m25.190630675s) [noise=true , address=145.79.227.16:4911 , piece_id=22JL4OCQ5R5XEWLGLGIWSMN3BTQV5XJUHRTA5WQ34I7JFQ3AEHDA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.320537405 (565h24m25.191620441s) [noise=true , address=145.79.227.42:4122 , piece_id=FSWIDVF6DUPUM57UWV5YFRQYEEYVEJGL74LMOED4BJDKELDDO2FA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.327901507 (565h24m25.198984553s) [noise=true , address=145.79.227.52:4446 , piece_id=ZGETDPCI52T24NGWWIJFPQAWSLYIKZRLUB7P7SUOQYMSBR5VNUFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.521949903 (565h24m25.393032948s) [noise=true , address=77.90.31.21:4877 , piece_id=KGT7ROBVSPR2FLAEVW247VKLXRMF3RALWSPIH6RM4RROVCRIARFQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.567183514 (565h24m25.438266549s) [noise=true , address=45.91.116.4:4939 , piece_id=WRQYALDRIHSTLIAO5DHVB4LKDYVUIIZ4FIXQLDURCERM7TQSMJHQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.584379547 (565h24m25.455462592s) [noise=true , address=77.90.31.19:4841 , piece_id=HAGL3UXSCOZVXOOJLQRNHOO3DTL5MSQICPMFHXODUTWTWU24YSCQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593753827 (565h24m25.464836862s) [noise=true , address=45.91.116.10:4081 , piece_id=4AT3SM6LNNMZNX3YPSWNE3ZL3T6JDBCVEZUYGXTCBXBGZX2JU4UQ, seg_pos={0 0}, upload=495727]: Done uploading piece. err: <nil> Dec 29 04:53:25.593774417 (565h24m25.464857452s) [seg_pos={0 0}, upload=495727]: Segment reached optimal threshold of 13 pieces. Dec 29 04:53:25.593991249 (565h24m25.465074284s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.593997611 (565h24m25.465080656s) [noise=true , address=45.91.116.13:4156 , piece_id=WKAMO7GNA3HCCXLXQINBAQVK3YP7EAYRADPCRD3LGY25GHKNVH6Q, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594179327 (565h24m25.465262362s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 29 04:53:25.594185318 (565h24m25.465268353s) [noise=true , address=145.79.227.40:4049 , piece_id=EXHR3FU6NGRO5DA4DLXHHCQCWKEBV4KKJYBZU4L3F34Z37BNNKWA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594389307 (565h24m25.465472362s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:60626->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 29 04:53:25.594406861 (565h24m25.465489906s) [noise=true , address=77.90.31.18:4805 , piece_id=P32D5K2AVV32NYPQQMU2OVZFZKR2LDGVR2FA5RC5ML3HS6IX5OTA, seg_pos={0 0}, upload=495727]: Long tail cancellation due to optimal threshold Dec 29 04:53:25.594449672 (565h24m25.465532737s) [seg_pos={0 0}, upload=495727]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 04:53:25.594545976 (565h24m25.465629011s) [upload=495727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 04:53:25.594551346 (565h24m25.465634391s) [upload=495727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 04:53:25.883092627 (565h24m25.754175662s) [upload=495727]: Done uploading segments... Dec 29 04:53:25.883111474 (565h24m25.754194519s) [upload=495727]: Upload finished. err: <nil> Duration: 30.003896848s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)