Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [XXX...X...XX.....XX..........X......XXX...X........X........] duration: 3.515328228s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:53:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515355329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.........X..X...............X.......X....X........] duration: 38.541834232s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:53:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 38.541854128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX.......] duration: 5.861580931s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:53:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 5.861608053s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.262480695s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:53:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.262521443s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.XXX...X...XX.....XX..........X......XXX...X........X.......] duration: 3.861054952s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:53:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.861083546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.........X..X...............X.......X....X.......] duration: 42.156953078s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 42.156974399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 10.207234183s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:52:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 10.207260834s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX......] duration: 5.81895229s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:52:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 5.818980423s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [..XXX...X...XX.....XX..........X......XXX...X........X......] duration: 3.552010968s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:52:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.552060812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [X..........X.........X..X...............X.......X....X......] duration: 44.258936541s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:52:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 44.258966897s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.646683224s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:51:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.646711388s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [...XXX...X...XX.....XX..........X......XXX...X........X.....] duration: 3.656405033s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:51:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.65648242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX.....] duration: 6.29174075s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:51:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 6.291761951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [XX..........X.........X..X...............X.......X....X.....] duration: 34.237992807s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:51:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 34.238018465s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....XXX...X...XX.....XX..........X......XXX...X........X....] duration: 4.018803514s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:51:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.018835434s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.645156214s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:50:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.645185591s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX....] duration: 6.021424622s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:50:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 6.021457394s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 10.184471538s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:50:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 10.184499972s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....XXX...X...XX.....XX..........X......XXX...X........X...] duration: 3.721731825s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:50:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.721763077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX...] duration: 6.081180448s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 6.081205245s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.XX..........X.........X..X...............X.......X....X....] duration: 15.502500051s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:49:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 15.502524928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 11.326390682s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:49:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 11.326425548s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [..XX..........X.........X..X...............X.......X....X...] duration: 23.754521301s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:49:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 23.754538964s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......XXX...X...XX.....XX..........X......XXX...X........X..] duration: 3.51274575s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:48:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.512807449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [...XX..........X.........X..X...............X.......X....X..] duration: 48.607862933s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:48:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 48.607877309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX..] duration: 6.080544086s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 6.080569735s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.588906226s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 6.588932075s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 13.557055295s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:48:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 13.557089482s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [....XX..........X.........X..X...............X.......X....X.] duration: 54.874002197s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:47:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 54.874017476s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX.] duration: 6.11079553s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:47:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 6.110813495s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......XXX...X...XX.....XX..........X......XXX...X........X.] duration: 4.621261818s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:47:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.621319138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 7.759715061s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:47:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.75973578s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........XXX...X...XX.....XX..........X......XXX...X........X] duration: 5.00143304s trace: https://jaeger.dev.storj.io/trace/d6cc88faa8cced4 FAIL (2025-12-26 16:47:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001459381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXXX] duration: 30.002987669s trace: https://jaeger.dev.storj.io/trace/7723dbb2684d3cf8 FAIL (2025-12-26 16:47:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A) context deadline exceeded ======LOGS====== 6 16:46:53.678772155 (505h17m53.549855201s) [upload=443393]: Starting upload Dec 26 16:46:53.678858690 (505h17m53.549941725s) [upload=443393]: Done starting upload Dec 26 16:46:53.678920939 (505h17m53.550003974s) [upload=443393]: Uploading segments... Dec 26 16:46:53.679071706 (505h17m53.550154741s) [upload=443393]: Got next segment. Inline: false Dec 26 16:46:53.679106472 (505h17m53.550189508s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:53.679110881 (505h17m53.550193916s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:53.927358718 (505h17m53.798441763s) [seg_pos={0 0}, upload=443393]: Begin upload segment... Dec 26 16:46:53.927414353 (505h17m53.798497399s) [seg_pos={0 0}, upload=443393]: Done begin upload segment. Dec 26 16:46:53.927445783 (505h17m53.798528828s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.927461082 (505h17m53.798544127s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928019415 (505h17m53.79910246s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928407605 (505h17m53.79949064s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.928798980 (505h17m53.799882025s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929172963 (505h17m53.800256008s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.929514302 (505h17m53.800597347s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.930056935 (505h17m53.801139971s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.931962264 (505h17m53.803045299s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.932171392 (505h17m53.803254437s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.933336019 (505h17m53.804419074s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.935343252 (505h17m53.806426297s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.937403254 (505h17m53.808486289s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939419153 (505h17m53.810502198s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.939797223 (505h17m53.810880258s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:53.941121354 (505h17m53.812204399s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Uploading piece... Dec 26 16:46:55.638067976 (505h17m55.509151011s) [upload=443393]: Next returned nil segment Dec 26 16:46:55.638092092 (505h17m55.509175137s) [upload=443393]: Waiting for error group managing segments... Dec 26 16:46:55.929283400 (505h17m55.800366446s) [noise=true , address=145.79.227.50:4374 , piece_id=VGWXWIVOUEAFYYRJP4QEQLD23D77N2QUE73MRCT2GW7ILU7COS5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.938300290 (505h17m55.809383335s) [noise=true , address=145.79.227.36:4517 , piece_id=74ZKMXOA7TREAQLZKGW6OBI4DGZCCJ5JDZUOOPDQ2JSLAIZ3OLMQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.941390795 (505h17m55.81247384s) [noise=true , address=145.79.227.42:4123 , piece_id=F4DRXTG344L3KNZYNUU4FYXKD7N2MMX2YSFOFQFI2WWP5RC7GLDQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.943202184 (505h17m55.814285229s) [noise=true , address=145.79.227.15:4876 , piece_id=MU7KWZSLRMG6LS5UGXURWS6ZXJCQKS4COIFGMOHPXIXYLYXTP5EQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.945121619 (505h17m55.816204664s) [noise=true , address=145.79.227.39:4013 , piece_id=VQN2IAKZUBC3344OCKYDE46CKZ7GJFAVLOQZ3WNHSLJKWW5BQX5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.958053675 (505h17m55.82913672s) [noise=true , address=145.79.227.6:4257 , piece_id=VM6AD2QGEXVIFOILFM3ZOPIUXF4X5WRNPL37SVPGQVCO5GIR3G6Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.959977357 (505h17m55.831060392s) [noise=true , address=145.79.227.48:4303 , piece_id=B25Z4DCXSV275453AD5L5N2J7GTAM7BVRAN3UCYZU43NSSZY4TSQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:55.966292213 (505h17m55.837375258s) [noise=true , address=145.79.227.47:4267 , piece_id=WUZ43YRNIDAH4OLKAYDCJET3XKD4AMTU5ODPA6FJ435PKWHXMCIA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.202880448 (505h17m56.073963483s) [noise=true , address=45.91.116.6:4974 , piece_id=DXDWFODAZ4H775HGTNTKD53DOBNS2GX7XOISA2OKJBSGDHYCBD5A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227006288 (505h17m56.098089323s) [noise=true , address=77.90.31.10:4519 , piece_id=L4CA2ULKVMUJTV5WJL27VDFPVEP3EOPSSQNPQVJUSAMWRRTF6OUA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.227815438 (505h17m56.098898473s) [noise=true , address=77.90.31.19:4842 , piece_id=5YN2XY73DWVLREOVCPZJHJMP634J22KNJVT25PLQHT7TVIQOF2AQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.244751667 (505h17m56.115834732s) [noise=true , address=45.91.116.7:4010 , piece_id=SRERBNLRO4AF7IQATEZC46JGESE7RLGUT6HQ6BLAD4DVNJFCO7MQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372793261 (505h17m56.243876296s) [noise=true , address=45.91.116.13:4156 , piece_id=CWZEQOR7KDBCX4Q7RWQ46TBTHC43XEGPK4SB66KC6L5VXFVGWBCA, seg_pos={0 0}, upload=443393]: Done uploading piece. err: <nil> Dec 26 16:46:56.372815163 (505h17m56.243898198s) [seg_pos={0 0}, upload=443393]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:56.373067634 (505h17m56.244150669s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373077332 (505h17m56.244160367s) [noise=true , address=45.91.116.12:4225 , piece_id=AMCBHACIFYH2RAZAPIIIYLHDCRGOFVBEEVXA2MFVKXYMGDEW2X4Q, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373142867 (505h17m56.244225902s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46866->77.90.31.15:4700: use of closed network connection Dec 26 16:46:56.373159598 (505h17m56.244242633s) [noise=true , address=77.90.31.15:4700 , piece_id=A526AK2MO4WTT3FAWYMZYFEMUOJ22UZJOAVCDYEJVTN7WQODQHCQ, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373377634 (505h17m56.244460679s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:56.373386871 (505h17m56.244469907s) [noise=true , address=45.91.116.5:4298 , piece_id=657CCRWTZS27FEFSL4BC7E5EMXTWTBNVWOTQXSHXONQ5OQA2PJ2A, seg_pos={0 0}, upload=443393]: Long tail cancellation due to optimal threshold Dec 26 16:46:56.373407912 (505h17m56.244490957s) [seg_pos={0 0}, upload=443393]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:56.373486912 (505h17m56.244569947s) [upload=443393]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:56.373536827 (505h17m56.244619862s) [upload=443393]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:56.671042289 (505h17m56.542125344s) [upload=443393]: Done uploading segments... Dec 26 16:46:56.671067307 (505h17m56.542150352s) [upload=443393]: Upload finished. err: <nil> Duration: 30.003002597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....XX..........X.........X..X...............X.......X....X] duration: 1m0.003532123s trace: https://jaeger.dev.storj.io/trace/109f3e067e010c2f FAIL (2025-12-26 16:47:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, Piece ID: DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q) context deadline exceeded; piecestore: (Node ID: 1ahTbggGRi6cJa4JPBfSEgLJYymrgVuLxSc15BV3qj5gPWPFiZ, Piece ID: WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA) context deadline exceeded; piecestore: (Node ID: 1m4M9WpEgS8BgLgQ2kwqzb6szQ2k4KMS3pv6DRQsEPS2ujPyLM, Piece ID: TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ) context deadline exceeded ======LOGS====== 26 16:46:05.282897453 (505h17m5.153980498s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.282954052 (505h17m5.154037097s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283314878 (505h17m5.154397923s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283493097 (505h17m5.154576143s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283720251 (505h17m5.154803296s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.283969956 (505h17m5.155053001s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284146873 (505h17m5.155229918s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284470589 (505h17m5.155553634s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284553537 (505h17m5.155636582s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.284974238 (505h17m5.156057283s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285009706 (505h17m5.156092751s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.285544504 (505h17m5.156627549s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.286800255 (505h17m5.15788337s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287403053 (505h17m5.158486098s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.287928925 (505h17m5.15901197s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:05.289150881 (505h17m5.160233926s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Uploading piece... Dec 26 16:46:06.768498316 (505h17m6.639581351s) [upload=443378]: Next returned nil segment Dec 26 16:46:06.768512754 (505h17m6.63959579s) [upload=443378]: Waiting for error group managing segments... Dec 26 16:46:07.177478726 (505h17m7.048561771s) [noise=true , address=145.79.227.11:4737 , piece_id=FAQBVVD3I2B5ZOPSR2VCZGSRQNVIRSZRVQCX5E33QECVEAOKLHRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.187360212 (505h17m7.058443247s) [noise=true , address=145.79.227.12:4797 , piece_id=OR6AQ5RKIULPNCDS4TXRRXNXKXZDM4TT2J367SRQJEYZAYYML6LA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.191564518 (505h17m7.062647563s) [noise=true , address=64.49.13.26:4950 , piece_id=NFO7SRXQGAUISAEPYC6OCLPCUTKFWR5PSOWUJZ5YHSLCYNZQQ7SQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.431132678 (505h17m7.302215713s) [noise=true , address=77.90.31.10:4530 , piece_id=DYASNSW6VYL6VFN2ZCYV6MWPLZA5KDMZRQF6ZKVSIG3RQPWESW7Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.443236505 (505h17m7.31431955s) [noise=true , address=77.90.31.15:4710 , piece_id=25MAPHYJKQV4XQ7X67IVTNXM2ZKPBALDEKK66ZK67TFX3M74SGRQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:07.515779830 (505h17m7.386862865s) [noise=true , address=77.90.31.21:4885 , piece_id=6YIKCRJQZ4Z2EKOB7Y7QZFVS2WIHV7CJUJVOKV5DS64CNLADLMVQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.030094386 (505h17m7.901177421s) [noise=true , address=145.79.227.40:4071 , piece_id=IBCGYA4LAJIRKFVJV2E6T2NPRCVADJIP5SAJ75H4QXDIMOQAN2NA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.108333867 (505h17m7.979416913s) [noise=true , address=77.90.31.16:4758 , piece_id=WBE5HZWKMO3AZSSHMJM7GXKVV66CD3MB24QFHQPWRY4GTIJFF2YA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.146456269 (505h17m8.017539314s) [noise=true , address=145.79.227.44:4199 , piece_id=7ZE6F5CL4VK6NKCPLM6H7RLIHLEO2RWEG5EGJFXF7X7Y4CSFX6VQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.167739325 (505h17m8.03882236s) [noise=true , address=77.90.31.18:4828 , piece_id=EBH3I5TADBPDMDRYQSG7NQVEFRJVNACURFHR3UTKQL2ZPE7KAFVA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.292396866 (505h17m8.163479911s) [noise=true , address=77.90.31.13:4644 , piece_id=6QLHFIXW7DIL3XRHVODFU6LPMRW25WSM77LRPYGVGUEN36NUALMA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.475687322 (505h17m8.346770367s) [noise=true , address=77.90.31.14:4685 , piece_id=4XZFFB7E642K5A4IKKN6Y6NJ42POJWUCIK7534PNM5A332YHTABA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.551777060 (505h17m8.422860105s) [noise=true , address=77.90.31.19:4860 , piece_id=P2FISE3LU4Q5ZVOOGYHZAO52MIAE5DYWLFY6CVXSZIUC5ONL7HYQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678298409 (505h17m8.549381454s) [noise=true , address=77.90.31.11:4578 , piece_id=TVND6IURSNSGQ4HEVP4552NVEVOYW3GFZQUM7KPR6K276JCCEJBQ, seg_pos={0 0}, upload=443378]: Done uploading piece. err: <nil> Dec 26 16:46:08.678323457 (505h17m8.549406502s) [seg_pos={0 0}, upload=443378]: Segment reached optimal threshold of 14 pieces. Dec 26 16:46:08.678549747 (505h17m8.549632793s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu): protocol: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39436->145.79.227.6:4271: use of closed network connection Dec 26 16:46:08.678558464 (505h17m8.549641509s) [noise=true , address=145.79.227.6:4271 , piece_id=AH24LB52VUPULWQ2YOQ7VDRYVXR72EPIZJ2CEIXKNEUIORD72EKA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678891048 (505h17m8.549974093s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ptydBd7T89noaveRPbmziYKKX9C1eNKaYG8BEfz2Zk4T9XtKz): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678907879 (505h17m8.549990924s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127aXyQbxPkETL9tjbKQodpMMteTiZjBQ5C4muNnXELbuy6H5TM): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.678919432 (505h17m8.550002467s) [noise=true , address=45.91.116.8:4055 , piece_id=6BWGZFJA6AY5EBGOD4PVRG63NG64HPVGBFHUBC767G5OE7YUVR3Q, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.678931354 (505h17m8.550014399s) [noise=true , address=45.91.116.9:4280 , piece_id=52WBQ3ODT6OXP35UPOSSNNZCYPJYREKAUASCBLNOXDOQXI7E6LFA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679190638 (505h17m8.550273714s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:08.679200026 (505h17m8.550283061s) [noise=true , address=64.49.13.140:4508 , piece_id=WKGTL6PEL62NGB4T3SZ4VH4YDG3523UJDLUCFRQRWJXVXDWXOMXA, seg_pos={0 0}, upload=443378]: Long tail cancellation due to optimal threshold Dec 26 16:46:08.679263507 (505h17m8.550346552s) [seg_pos={0 0}, upload=443378]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:46:08.679352386 (505h17m8.550435431s) [upload=443378]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:08.679359249 (505h17m8.550442294s) [upload=443378]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:08.985467612 (505h17m8.856550657s) [upload=443378]: Done uploading segments... Dec 26 16:46:08.985487951 (505h17m8.856570997s) [upload=443378]: Upload finished. err: <nil> Duration: 1m0.00356145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XXX] duration: 30.00279331s trace: https://jaeger.dev.storj.io/trace/1cd89be71e0fb7d5 FAIL (2025-12-26 16:46:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: ZVRXPJ2YR2NZQFWLKBHXUQGR7WIR4ELCQC6WEB3MHZHVPP2XQTJQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: RFPGHKXQWWF2XYVLJ3X7VRR4CMFKCTNSBY27J6ZWC7VCZNVTCSGA) context deadline exceeded ======LOGS====== Dec 26 16:46:23.451796754 (505h17m23.32287979s) [upload=443385]: Starting upload Dec 26 16:46:23.451852972 (505h17m23.322936007s) [upload=443385]: Done starting upload Dec 26 16:46:23.451902466 (505h17m23.322985501s) [upload=443385]: Uploading segments... Dec 26 16:46:23.452022916 (505h17m23.323105951s) [upload=443385]: Got next segment. Inline: false Dec 26 16:46:23.452049036 (505h17m23.323132071s) [upload=443385]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:46:23.452053394 (505h17m23.323136429s) [upload=443385]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:46:23.706619552 (505h17m23.577702617s) [seg_pos={0 0}, upload=443385]: Begin upload segment... Dec 26 16:46:23.706676992 (505h17m23.577760047s) [seg_pos={0 0}, upload=443385]: Done begin upload segment. Dec 26 16:46:23.706757164 (505h17m23.577840199s) [noise=true , address=145.79.227.2:4114 , piece_id=VZBLEFWGZMU26RGWB664ABU3BWZIOTIMH4DQXKGHHETSZP2K7KUA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.706770299 (505h17m23.577853344s) [noise=true , address=45.91.116.4:4937 , piece_id=X5ZOL7QTENXPRJEMQISFSLBTJVITOWQ5FXZRZJMRD24TZHKHBQHQ, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.707531638 (505h17m23.578614673s) [noise=true , address=77.90.31.13:4627 , piece_id=6PMFSC7TLBB4EWEN46G5ICQ2WVKT7TD6WCIVOAIVUTHTF3C6EXSQ, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.707750416 (505h17m23.578833461s) [noise=true , address=145.79.227.36:4517 , piece_id=MTC3DVFCS3OX4DMDDW42SGNHDU4VQBGCNHHE23V34S3NPGUO672Q, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.708130489 (505h17m23.579213535s) [noise=true , address=77.90.31.11:4555 , piece_id=MC3ROQMNWROCQHKRHRCXWQHCW3VJES4NPTQDPRBST6M3WDMX4DZA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.708326081 (505h17m23.579409136s) [noise=true , address=145.79.227.15:4876 , piece_id=QVWZDRM6VH4EJPIMHB5M7XH72E2S5HGO3VUARE3FEIERBGJ6AFKA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.708653946 (505h17m23.579736991s) [noise=true , address=145.79.227.39:4013 , piece_id=BL4DRD4RYKF24C6BBOQFLTAANI36C63HXELILBNWBELTOPMXFYRA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.708985367 (505h17m23.580068422s) [noise=true , address=145.79.227.22:4511 , piece_id=D2CARBBRJH3L6FNCTDGQPC6VZ5QFFARNLJBO7OOC5Z4I5MZUROFQ, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.709131615 (505h17m23.58021466s) [noise=true , address=45.91.116.10:4084 , piece_id=MYXGLVIO4V7NPQ6OZYITMXWXDJZPWYU244A256PVJHGW67KUBITA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.709507391 (505h17m23.580590436s) [noise=true , address=45.91.116.9:4261 , piece_id=ZVRXPJ2YR2NZQFWLKBHXUQGR7WIR4ELCQC6WEB3MHZHVPP2XQTJQ, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.709618713 (505h17m23.580701748s) [noise=true , address=145.79.227.13:4803 , piece_id=NZEDNHKJLUA6WMQPUCJOWJZ6NSDZMJDBJFHVXPJBKRR6STZVZZ3A, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.710088027 (505h17m23.581171082s) [noise=true , address=45.91.116.7:4010 , piece_id=RFPGHKXQWWF2XYVLJ3X7VRR4CMFKCTNSBY27J6ZWC7VCZNVTCSGA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.710613227 (505h17m23.581696272s) [noise=true , address=145.79.227.42:4121 , piece_id=DNAV3SRJYPWBFI4RISJ6AGDKYLGZWLHCBSZE6SPWT3BWS6YZY5OA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.711079113 (505h17m23.582162179s) [noise=true , address=45.91.116.8:4045 , piece_id=YWGYIXSWHOQT4FKULF2O4SWFHTZN4JR6AXRR5BHV5DBK37CSUDPA, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.711762996 (505h17m23.582846041s) [noise=true , address=145.79.227.16:4909 , piece_id=NZC26MTVTAX3MWCXBGDZ6YKP34FMKUVHSSKMEC6A55AEDXEF4JCQ, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:23.712571195 (505h17m23.58365423s) [noise=true , address=145.79.227.49:4339 , piece_id=UEHKSXXTDDBKFG6RJY4LP7CU5OPTDCFFMG5EQ5NN5KXMLXDXYACQ, seg_pos={0 0}, upload=443385]: Uploading piece... Dec 26 16:46:25.376747737 (505h17m25.247830772s) [upload=443385]: Next returned nil segment Dec 26 16:46:25.376778145 (505h17m25.24786118s) [upload=443385]: Waiting for error group managing segments... Dec 26 16:46:25.713814241 (505h17m25.584897286s) [noise=true , address=145.79.227.49:4339 , piece_id=UEHKSXXTDDBKFG6RJY4LP7CU5OPTDCFFMG5EQ5NN5KXMLXDXYACQ, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.713899162 (505h17m25.584982207s) [noise=true , address=145.79.227.36:4517 , piece_id=MTC3DVFCS3OX4DMDDW42SGNHDU4VQBGCNHHE23V34S3NPGUO672Q, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.714316427 (505h17m25.585399462s) [noise=true , address=145.79.227.15:4876 , piece_id=QVWZDRM6VH4EJPIMHB5M7XH72E2S5HGO3VUARE3FEIERBGJ6AFKA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.716661582 (505h17m25.587744627s) [noise=true , address=145.79.227.42:4121 , piece_id=DNAV3SRJYPWBFI4RISJ6AGDKYLGZWLHCBSZE6SPWT3BWS6YZY5OA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.730295433 (505h17m25.601378478s) [noise=true , address=145.79.227.22:4511 , piece_id=D2CARBBRJH3L6FNCTDGQPC6VZ5QFFARNLJBO7OOC5Z4I5MZUROFQ, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.733410455 (505h17m25.60449349s) [noise=true , address=145.79.227.2:4114 , piece_id=VZBLEFWGZMU26RGWB664ABU3BWZIOTIMH4DQXKGHHETSZP2K7KUA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.744587538 (505h17m25.615670574s) [noise=true , address=145.79.227.39:4013 , piece_id=BL4DRD4RYKF24C6BBOQFLTAANI36C63HXELILBNWBELTOPMXFYRA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.777242870 (505h17m25.648325915s) [noise=true , address=145.79.227.13:4803 , piece_id=NZEDNHKJLUA6WMQPUCJOWJZ6NSDZMJDBJFHVXPJBKRR6STZVZZ3A, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:25.908437527 (505h17m25.779520572s) [noise=true , address=145.79.227.16:4909 , piece_id=NZC26MTVTAX3MWCXBGDZ6YKP34FMKUVHSSKMEC6A55AEDXEF4JCQ, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:26.083001197 (505h17m25.954084232s) [noise=true , address=45.91.116.9:4261 , piece_id=ZVRXPJ2YR2NZQFWLKBHXUQGR7WIR4ELCQC6WEB3MHZHVPP2XQTJQ, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:26.102726176 (505h17m25.973809211s) [noise=true , address=45.91.116.8:4045 , piece_id=YWGYIXSWHOQT4FKULF2O4SWFHTZN4JR6AXRR5BHV5DBK37CSUDPA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:26.659409559 (505h17m26.530492594s) [noise=true , address=45.91.116.7:4010 , piece_id=RFPGHKXQWWF2XYVLJ3X7VRR4CMFKCTNSBY27J6ZWC7VCZNVTCSGA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:26.713334832 (505h17m26.584417867s) [noise=true , address=45.91.116.4:4937 , piece_id=X5ZOL7QTENXPRJEMQISFSLBTJVITOWQ5FXZRZJMRD24TZHKHBQHQ, seg_pos={0 0}, upload=443385]: Done uploading piece. err: <nil> Dec 26 16:46:26.713358407 (505h17m26.584441442s) [seg_pos={0 0}, upload=443385]: Segment reached optimal threshold of 13 pieces. Dec 26 16:46:26.713713162 (505h17m26.584796207s) [noise=true , address=45.91.116.10:4084 , piece_id=MYXGLVIO4V7NPQ6OZYITMXWXDJZPWYU244A256PVJHGW67KUBITA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:26.713720416 (505h17m26.584803451s) [noise=true , address=45.91.116.10:4084 , piece_id=MYXGLVIO4V7NPQ6OZYITMXWXDJZPWYU244A256PVJHGW67KUBITA, seg_pos={0 0}, upload=443385]: Long tail cancellation due to optimal threshold Dec 26 16:46:26.713922411 (505h17m26.585005446s) [noise=true , address=77.90.31.11:4555 , piece_id=MC3ROQMNWROCQHKRHRCXWQHCW3VJES4NPTQDPRBST6M3WDMX4DZA, seg_pos={0 0}, upload=443385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 26 16:46:26.713928272 (505h17m26.585011307s) [noise=true , address=77.90.31.11:4555 , piece_id=MC3ROQMNWROCQHKRHRCXWQHCW3VJES4NPTQDPRBST6M3WDMX4DZA, seg_pos={0 0}, upload=443385]: Long tail cancellation due to optimal threshold Dec 26 16:46:26.714108796 (505h17m26.585191831s) [noise=true , address=77.90.31.13:4627 , piece_id=6PMFSC7TLBB4EWEN46G5ICQ2WVKT7TD6WCIVOAIVUTHTF3C6EXSQ, seg_pos={0 0}, upload=443385]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:55286->77.90.31.13:4627: use of closed network connection; conn closed: op Dec 26 16:46:26.714117222 (505h17m26.585200267s) [noise=true , address=77.90.31.13:4627 , piece_id=6PMFSC7TLBB4EWEN46G5ICQ2WVKT7TD6WCIVOAIVUTHTF3C6EXSQ, seg_pos={0 0}, upload=443385]: Long tail cancellation due to optimal threshold Dec 26 16:46:26.714134063 (505h17m26.585217099s) [seg_pos={0 0}, upload=443385]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:46:26.714202764 (505h17m26.585285799s) [upload=443385]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:46:26.714207323 (505h17m26.585290358s) [upload=443385]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:46:27.015364008 (505h17m26.886447043s) [upload=443385]: Done uploading segments... Dec 26 16:46:27.015389457 (505h17m26.886472492s) [upload=443385]: Upload finished. err: <nil> Duration: 30.002816783s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.716624489s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 5.716648005s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.........XXX...X...XX.....XX..........X......XXX...X........] duration: 3.558043204s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:46:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.558069845s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 7.079222063s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:46:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.079251187s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...XX] duration: 30.003469434s trace: https://jaeger.dev.storj.io/trace/411eda7a19a37531 FAIL (2025-12-26 16:46:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 16:45:48.426197459 (505h16m48.297280494s) [upload=443374]: Starting upload Dec 26 16:45:48.426264787 (505h16m48.297347822s) [upload=443374]: Done starting upload Dec 26 16:45:48.426557484 (505h16m48.297640519s) [upload=443374]: Uploading segments... Dec 26 16:45:48.426673465 (505h16m48.2977565s) [upload=443374]: Got next segment. Inline: false Dec 26 16:45:48.426713812 (505h16m48.297796848s) [upload=443374]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:45:48.426720305 (505h16m48.29780335s) [upload=443374]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:45:48.669938244 (505h16m48.541021309s) [seg_pos={0 0}, upload=443374]: Begin upload segment... Dec 26 16:45:48.670005542 (505h16m48.541088577s) [seg_pos={0 0}, upload=443374]: Done begin upload segment. Dec 26 16:45:48.670091226 (505h16m48.541174261s) [noise=true , address=145.79.227.12:4767 , piece_id=HWVIEDUXPPL4ZNYBMUWWYEOKYYL2G4N277SLNX7VVMQCHOZIPEQA, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.670126543 (505h16m48.541209588s) [noise=true , address=145.79.227.11:4729 , piece_id=CCGUDFRW5DDBLWZWZPJ3DE5RTKPUS4456KRHKUYM5M4UYSCM3QLQ, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.670633307 (505h16m48.541716342s) [noise=true , address=145.79.227.18:4007 , piece_id=3KK2RF6YCESKLTDMUHVVKD5NPGSPD7RBMHAGLLQIBZAZ5FQ7WVYA, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.670763916 (505h16m48.541846951s) [noise=true , address=77.90.31.21:4880 , piece_id=J5IXIBHDZEWWJJSRIAYY2Q3Q6TF5UEQWCQZ46C6KAYOAQO6UDQ5Q, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.671075259 (505h16m48.542158294s) [noise=true , address=145.79.227.29:4942 , piece_id=7UMOVTBC5S2SBR4PVO2WQQ56PA52BKJ64NH2KY27GKC6DLOATN6Q, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.671332158 (505h16m48.542415193s) [noise=true , address=145.79.227.43:4158 , piece_id=GYGMPUE36TX3WSF23KOBT2IPQXOQPRMEEPZDYHAYRC2WTT6645DQ, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.671586231 (505h16m48.542669267s) [noise=true , address=77.90.31.11:4554 , piece_id=6PTKGOLXG4LLPJK4SL7GQO6YYEEN3JU2AUHT2B34OFZ4V6GTRE7Q, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.671820478 (505h16m48.542903523s) [noise=true , address=45.91.116.4:4940 , piece_id=SI457GIWHFLW4HTS7YOH5N2USV36YYJXD5DGQOXKYKU44A3LBQPA, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.672091554 (505h16m48.543174589s) [noise=true , address=45.91.116.6:4973 , piece_id=2VI2LR6HBIV6VXIDP6YSQ276RB2VL2GUCME2MCBFTIFNHRNZLIFQ, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.672373611 (505h16m48.543456646s) [noise=true , address=145.79.227.52:4446 , piece_id=RYVE3GRKBBHVJ3BU24VU3O66KKHBQ66DXMZE6HCREWZQSPW6GLOQ, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.672846622 (505h16m48.543929657s) [noise=true , address=145.79.227.21:4475 , piece_id=7MS6EMSC3CSG24LBNRANV4ND4O2SD6WA3LS5DHXG5MRLNIUGCQFA, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.673196789 (505h16m48.544279824s) [noise=true , address=145.79.227.37:4565 , piece_id=QV5WTWKEPS55AJCIUPOGWXX6QJ27BS32SK3JF6TYQL6TLQGIIR2Q, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.673482954 (505h16m48.544565989s) [noise=true , address=145.79.227.36:4518 , piece_id=AXN4CURG2BYPMJSCKTZICLGWM4XY3F5VWJRZKHDFRGBPPT2XZJ6Q, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.674168660 (505h16m48.545251695s) [noise=true , address=145.79.227.45:4231 , piece_id=D4VCI74C526UO4VYPCTMJF6UBR7KPRDJWYY45KXSYK5W6HL35NBQ, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.675150950 (505h16m48.546233995s) [noise=true , address=145.79.227.13:4803 , piece_id=AWTJD26DVN6ULT3CORSDWI6CNBEIULQCSFZEW3HVFN4YJ5NERDOQ, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:48.675973375 (505h16m48.54705642s) [noise=true , address=145.79.227.49:4337 , piece_id=WLXEAWA6IF7XIXH5BNPVWJUXSXNIGDOUDHONEB2VOGRZDWQV2OAA, seg_pos={0 0}, upload=443374]: Uploading piece... Dec 26 16:45:49.896837744 (505h16m49.767920789s) [noise=true , address=77.90.31.11:4554 , piece_id=6PTKGOLXG4LLPJK4SL7GQO6YYEEN3JU2AUHT2B34OFZ4V6GTRE7Q, seg_pos={0 0}, upload=443374]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: storage node overloaded, request limit: 2000 Dec 26 16:45:50.286761106 (505h16m50.157844142s) [upload=443374]: Next returned nil segment Dec 26 16:45:50.286776545 (505h16m50.157859581s) [upload=443374]: Waiting for error group managing segments... Dec 26 16:45:50.676231897 (505h16m50.547314942s) [noise=true , address=145.79.227.37:4565 , piece_id=QV5WTWKEPS55AJCIUPOGWXX6QJ27BS32SK3JF6TYQL6TLQGIIR2Q, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.677635088 (505h16m50.548718134s) [noise=true , address=145.79.227.21:4475 , piece_id=7MS6EMSC3CSG24LBNRANV4ND4O2SD6WA3LS5DHXG5MRLNIUGCQFA, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.678172852 (505h16m50.549255897s) [noise=true , address=145.79.227.36:4518 , piece_id=AXN4CURG2BYPMJSCKTZICLGWM4XY3F5VWJRZKHDFRGBPPT2XZJ6Q, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.678849932 (505h16m50.549932987s) [noise=true , address=145.79.227.11:4729 , piece_id=CCGUDFRW5DDBLWZWZPJ3DE5RTKPUS4456KRHKUYM5M4UYSCM3QLQ, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.680150177 (505h16m50.551233222s) [noise=true , address=145.79.227.49:4337 , piece_id=WLXEAWA6IF7XIXH5BNPVWJUXSXNIGDOUDHONEB2VOGRZDWQV2OAA, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.690101276 (505h16m50.561184321s) [noise=true , address=145.79.227.29:4942 , piece_id=7UMOVTBC5S2SBR4PVO2WQQ56PA52BKJ64NH2KY27GKC6DLOATN6Q, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.693726209 (505h16m50.564809254s) [noise=true , address=145.79.227.13:4803 , piece_id=AWTJD26DVN6ULT3CORSDWI6CNBEIULQCSFZEW3HVFN4YJ5NERDOQ, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.699363603 (505h16m50.570446638s) [noise=true , address=145.79.227.45:4231 , piece_id=D4VCI74C526UO4VYPCTMJF6UBR7KPRDJWYY45KXSYK5W6HL35NBQ, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.706270275 (505h16m50.57735333s) [noise=true , address=145.79.227.18:4007 , piece_id=3KK2RF6YCESKLTDMUHVVKD5NPGSPD7RBMHAGLLQIBZAZ5FQ7WVYA, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.719249149 (505h16m50.590332194s) [noise=true , address=145.79.227.52:4446 , piece_id=RYVE3GRKBBHVJ3BU24VU3O66KKHBQ66DXMZE6HCREWZQSPW6GLOQ, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.752050669 (505h16m50.623133704s) [noise=true , address=145.79.227.12:4767 , piece_id=HWVIEDUXPPL4ZNYBMUWWYEOKYYL2G4N277SLNX7VVMQCHOZIPEQA, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.916068758 (505h16m50.787151793s) [noise=true , address=45.91.116.6:4973 , piece_id=2VI2LR6HBIV6VXIDP6YSQ276RB2VL2GUCME2MCBFTIFNHRNZLIFQ, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.960611716 (505h16m50.831694751s) [noise=true , address=77.90.31.21:4880 , piece_id=J5IXIBHDZEWWJJSRIAYY2Q3Q6TF5UEQWCQZ46C6KAYOAQO6UDQ5Q, seg_pos={0 0}, upload=443374]: Done uploading piece. err: <nil> Dec 26 16:45:50.960627886 (505h16m50.831710921s) [seg_pos={0 0}, upload=443374]: Segment reached optimal threshold of 13 pieces. Dec 26 16:45:50.960846332 (505h16m50.831929367s) [noise=true , address=45.91.116.4:4940 , piece_id=SI457GIWHFLW4HTS7YOH5N2USV36YYJXD5DGQOXKYKU44A3LBQPA, seg_pos={0 0}, upload=443374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 16:45:50.960852744 (505h16m50.831935779s) [noise=true , address=45.91.116.4:4940 , piece_id=SI457GIWHFLW4HTS7YOH5N2USV36YYJXD5DGQOXKYKU44A3LBQPA, seg_pos={0 0}, upload=443374]: Long tail cancellation due to optimal threshold Dec 26 16:45:50.961076180 (505h16m50.832159225s) [noise=true , address=145.79.227.43:4158 , piece_id=GYGMPUE36TX3WSF23KOBT2IPQXOQPRMEEPZDYHAYRC2WTT6645DQ, seg_pos={0 0}, upload=443374]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 16:45:50.961082161 (505h16m50.832165196s) [noise=true , address=145.79.227.43:4158 , piece_id=GYGMPUE36TX3WSF23KOBT2IPQXOQPRMEEPZDYHAYRC2WTT6645DQ, seg_pos={0 0}, upload=443374]: Long tail cancellation due to optimal threshold Dec 26 16:45:50.961096568 (505h16m50.832179603s) [seg_pos={0 0}, upload=443374]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:45:50.961175810 (505h16m50.832258845s) [upload=443374]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:45:50.961180959 (505h16m50.832263994s) [upload=443374]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:45:51.254663902 (505h16m51.125746957s) [upload=443374]: Done uploading segments... Dec 26 16:45:51.254690132 (505h16m51.125773177s) [upload=443374]: Upload finished. err: <nil> Duration: 30.003492719s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 16.390738774s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:45:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 16.390765925s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..........XXX...X...XX.....XX..........X......XXX...X.......] duration: 4.430752836s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:45:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.430864919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...X] duration: 30.002295901s trace: https://jaeger.dev.storj.io/trace/4b6d901e1cb9803 FAIL (2025-12-26 16:45:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: FSOCSDNJ2MZEFRT6DRQZVL2KZCXREM2RCH2QNR2EFVPUOGBJXKGA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: HXUXZDUZRTC3OFCAU7PNRLYNWYIKBSXERAIZAQCCGTAMQHFVJENA) context deadline exceeded ======LOGS====== ec 26 16:45:08.677958112 (505h16m8.549041187s) [upload=443365]: Done starting upload Dec 26 16:45:08.678205975 (505h16m8.54928902s) [upload=443365]: Uploading segments... Dec 26 16:45:08.678279885 (505h16m8.549362921s) [upload=443365]: Got next segment. Inline: false Dec 26 16:45:08.678317136 (505h16m8.549400181s) [upload=443365]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:45:08.678323428 (505h16m8.549406473s) [upload=443365]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:45:08.922651621 (505h16m8.793734666s) [seg_pos={0 0}, upload=443365]: Begin upload segment... Dec 26 16:45:08.922695354 (505h16m8.7937784s) [seg_pos={0 0}, upload=443365]: Done begin upload segment. Dec 26 16:45:08.922733126 (505h16m8.793816161s) [noise=true , address=145.79.227.13:4803 , piece_id=3LQVF2GKQAKM6EFRF4XVVNTFLO47564WLEBTYOG6J6XKZSMH7WSA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.922879155 (505h16m8.79396219s) [noise=true , address=77.90.31.18:4808 , piece_id=FSOCSDNJ2MZEFRT6DRQZVL2KZCXREM2RCH2QNR2EFVPUOGBJXKGA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.923326376 (505h16m8.794409431s) [noise=true , address=145.79.227.41:4087 , piece_id=EVONMXSPEMCFF7YEM4KUYICZQUX4Y5G7L67KU4NRFLJ6JG5W6XWQ, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.923749973 (505h16m8.794833028s) [noise=true , address=145.79.227.37:4566 , piece_id=FTZ2JEJ6JP55MNGSIZLVKG4WVR5XF3M2HHIEDTCEAPSI3HBGA4KA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.924172047 (505h16m8.795255082s) [noise=true , address=145.79.227.45:4231 , piece_id=PTRRVELL5ZTOJSOWUTGA7ZREYSH6AANIUBQPUIKDP6KPO35NS47A, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.924597707 (505h16m8.795680742s) [noise=true , address=77.90.31.17:4770 , piece_id=7NLYO3Z7IUX2AHITQK6FL3QFFPKQPMDKV5ZHNYRK2ZUS4PKSNPGA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.925054427 (505h16m8.796137472s) [noise=true , address=145.79.227.50:4374 , piece_id=ZLH4MGFBMUJETJJYA6ULKH24HOT2EZ4IWAN4OJTK3JBQCVLBE4BA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.925512429 (505h16m8.796595474s) [noise=true , address=145.79.227.49:4337 , piece_id=SSOORH3FGNVVJGAXBBAMRSREFCPXKRU7QSVG34YEJZCHWBYZ4VEA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.927176798 (505h16m8.798259833s) [noise=true , address=45.91.116.12:4225 , piece_id=HXUXZDUZRTC3OFCAU7PNRLYNWYIKBSXERAIZAQCCGTAMQHFVJENA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.927762043 (505h16m8.798845088s) [noise=true , address=145.79.227.36:4518 , piece_id=RCU5SNIQZYIKWXCJMUTP6XPOOMXY43SOUMV6SVC35RMHZWRINGDQ, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.929319188 (505h16m8.800402233s) [noise=true , address=77.90.31.14:4663 , piece_id=A3IP5LKOXU5JHCNMONDC5SIC2LGFRUAKUJQTJOMIYYZJRV3KSCYA, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.930605137 (505h16m8.801688182s) [noise=true , address=145.79.227.29:4943 , piece_id=O3AQEDFRDPXHCO6VM7SZW3PCSW4Q3WMBVUTEASW5GNAYMRQEB6TQ, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.931710962 (505h16m8.802793997s) [noise=true , address=145.79.227.48:4303 , piece_id=QBMZIKBOZLT6VY5ACOLHLMOPKNOJCB5EXFQA3FQEE7CRIGK32FGQ, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.932742366 (505h16m8.803825411s) [noise=true , address=145.79.227.16:4911 , piece_id=K6Y3CAIMVSWO22EZII2ICKSVYAIGCYR7FQ73ELH5UQZRUXMCQEWQ, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.934412306 (505h16m8.805495341s) [noise=true , address=45.91.116.5:4300 , piece_id=VFRAEVF4BLHQF6VONIP54BJEPSOD2FAONMR3LRLL6JELMEFEYO4Q, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:08.936033974 (505h16m8.807117009s) [noise=true , address=145.79.227.22:4509 , piece_id=TBFSXOVK2JBBJAV6DNN7JIRUQ3KHCSGGWU5SMOSPMMV2J2XBT6DQ, seg_pos={0 0}, upload=443365]: Uploading piece... Dec 26 16:45:10.613012503 (505h16m10.484095558s) [upload=443365]: Next returned nil segment Dec 26 16:45:10.613038543 (505h16m10.484121588s) [upload=443365]: Waiting for error group managing segments... Dec 26 16:45:10.922433064 (505h16m10.793516109s) [noise=true , address=145.79.227.37:4566 , piece_id=FTZ2JEJ6JP55MNGSIZLVKG4WVR5XF3M2HHIEDTCEAPSI3HBGA4KA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.923665050 (505h16m10.794748085s) [noise=true , address=145.79.227.50:4374 , piece_id=ZLH4MGFBMUJETJJYA6ULKH24HOT2EZ4IWAN4OJTK3JBQCVLBE4BA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.934999383 (505h16m10.806082418s) [noise=true , address=145.79.227.45:4231 , piece_id=PTRRVELL5ZTOJSOWUTGA7ZREYSH6AANIUBQPUIKDP6KPO35NS47A, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.935911359 (505h16m10.806994394s) [noise=true , address=145.79.227.41:4087 , piece_id=EVONMXSPEMCFF7YEM4KUYICZQUX4Y5G7L67KU4NRFLJ6JG5W6XWQ, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.938514236 (505h16m10.809597271s) [noise=true , address=145.79.227.36:4518 , piece_id=RCU5SNIQZYIKWXCJMUTP6XPOOMXY43SOUMV6SVC35RMHZWRINGDQ, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.948597245 (505h16m10.81968028s) [noise=true , address=145.79.227.13:4803 , piece_id=3LQVF2GKQAKM6EFRF4XVVNTFLO47564WLEBTYOG6J6XKZSMH7WSA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.948850668 (505h16m10.819933713s) [noise=true , address=145.79.227.49:4337 , piece_id=SSOORH3FGNVVJGAXBBAMRSREFCPXKRU7QSVG34YEJZCHWBYZ4VEA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.953149984 (505h16m10.824233029s) [noise=true , address=145.79.227.48:4303 , piece_id=QBMZIKBOZLT6VY5ACOLHLMOPKNOJCB5EXFQA3FQEE7CRIGK32FGQ, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.957098552 (505h16m10.828181598s) [noise=true , address=145.79.227.22:4509 , piece_id=TBFSXOVK2JBBJAV6DNN7JIRUQ3KHCSGGWU5SMOSPMMV2J2XBT6DQ, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:10.959969420 (505h16m10.831052455s) [noise=true , address=145.79.227.29:4943 , piece_id=O3AQEDFRDPXHCO6VM7SZW3PCSW4Q3WMBVUTEASW5GNAYMRQEB6TQ, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:11.007142766 (505h16m10.878225801s) [noise=true , address=145.79.227.16:4911 , piece_id=K6Y3CAIMVSWO22EZII2ICKSVYAIGCYR7FQ73ELH5UQZRUXMCQEWQ, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:11.190985294 (505h16m11.062068329s) [noise=true , address=77.90.31.18:4808 , piece_id=FSOCSDNJ2MZEFRT6DRQZVL2KZCXREM2RCH2QNR2EFVPUOGBJXKGA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:11.203343827 (505h16m11.074426872s) [noise=true , address=45.91.116.12:4225 , piece_id=HXUXZDUZRTC3OFCAU7PNRLYNWYIKBSXERAIZAQCCGTAMQHFVJENA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: <nil> Dec 26 16:45:11.203371139 (505h16m11.074454184s) [seg_pos={0 0}, upload=443365]: Segment reached optimal threshold of 13 pieces. Dec 26 16:45:11.203763116 (505h16m11.074846161s) [noise=true , address=45.91.116.5:4300 , piece_id=VFRAEVF4BLHQF6VONIP54BJEPSOD2FAONMR3LRLL6JELMEFEYO4Q, seg_pos={0 0}, upload=443365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 16:45:11.203773375 (505h16m11.07485642s) [noise=true , address=45.91.116.5:4300 , piece_id=VFRAEVF4BLHQF6VONIP54BJEPSOD2FAONMR3LRLL6JELMEFEYO4Q, seg_pos={0 0}, upload=443365]: Long tail cancellation due to optimal threshold Dec 26 16:45:11.203918371 (505h16m11.075001416s) [noise=true , address=77.90.31.14:4663 , piece_id=A3IP5LKOXU5JHCNMONDC5SIC2LGFRUAKUJQTJOMIYYZJRV3KSCYA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: write tcp 188.166.225.170:36182->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:36182->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36182->77.90.31.14:4663: use of closed network connection Dec 26 16:45:11.203927369 (505h16m11.075010424s) [noise=true , address=77.90.31.14:4663 , piece_id=A3IP5LKOXU5JHCNMONDC5SIC2LGFRUAKUJQTJOMIYYZJRV3KSCYA, seg_pos={0 0}, upload=443365]: Long tail cancellation due to optimal threshold Dec 26 16:45:11.203932318 (505h16m11.075015373s) [noise=true , address=77.90.31.17:4770 , piece_id=7NLYO3Z7IUX2AHITQK6FL3QFFPKQPMDKV5ZHNYRK2ZUS4PKSNPGA, seg_pos={0 0}, upload=443365]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: read tcp 188.166.225.170:57962->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 16:45:11.203939592 (505h16m11.075022627s) [noise=true , address=77.90.31.17:4770 , piece_id=7NLYO3Z7IUX2AHITQK6FL3QFFPKQPMDKV5ZHNYRK2ZUS4PKSNPGA, seg_pos={0 0}, upload=443365]: Long tail cancellation due to optimal threshold Dec 26 16:45:11.203959288 (505h16m11.075042344s) [seg_pos={0 0}, upload=443365]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:45:11.204115136 (505h16m11.075198201s) [upload=443365]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:45:11.204125696 (505h16m11.075208751s) [upload=443365]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:45:11.499375092 (505h16m11.370458137s) [upload=443365]: Done uploading segments... Dec 26 16:45:11.499396172 (505h16m11.370479217s) [upload=443365]: Upload finished. err: <nil> Duration: 30.002322141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [......XX..........X.........X..X...............X.......X....] duration: 33.596590434s trace: https://jaeger.dev.storj.io/trace/14e86a12188d224b FAIL (2025-12-26 16:45:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 126oNZ8ZQpENsHFQy2RCab3g9eNJ1JBVqxNkcB7mMCXTftap84W, Piece ID: 6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ) context deadline exceeded; piecestore: (Node ID: 12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi, Piece ID: 44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA) context deadline exceeded; piecestore: (Node ID: 1WsJHk8jYeQAg4G4YXwCTjUSG4cX9xYhrZFmVpFQ5brFVz6TU5, Piece ID: BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA) context deadline exceeded ======LOGS====== 6 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639224730 (505h11m56.510307775s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.639702510 (505h11m56.510785555s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640116127 (505h11m56.511199173s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640171784 (505h11m56.511254829s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640518985 (505h11m56.51160204s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640819537 (505h11m56.511902582s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.640975103 (505h11m56.512058149s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641317956 (505h11m56.512401002s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.641413087 (505h11m56.512496132s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642007388 (505h11m56.513090434s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.642606971 (505h11m56.513690016s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643110820 (505h11m56.514193865s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.643616243 (505h11m56.514699288s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:56.644160849 (505h11m56.515243894s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Uploading piece... Dec 26 16:40:57.909019390 (505h11m57.780102425s) [upload=443308]: Next returned nil segment Dec 26 16:40:57.909046512 (505h11m57.780129547s) [upload=443308]: Waiting for error group managing segments... Dec 26 16:40:58.279921182 (505h11m58.151004217s) [noise=true , address=64.49.13.27:4965 , piece_id=JNYCPACXREVCJYKSXHEUAFEGGEI5QPW4VSWLJFLPSLUTT65AYLXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.509473179 (505h11m58.380556214s) [noise=true , address=145.79.227.43:4175 , piece_id=T63E5RK7U3N7GHEAOQAFHCV5UPWPC2RNGBCTKD7XB5ODY44P5TMQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.526249174 (505h11m58.397332209s) [noise=true , address=145.79.227.17:4978 , piece_id=JDFCSH37HLE7RXUJVIC4HU2TNDVNPCKFWFUNOL7AZCALMIPZJWPA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.534222977 (505h11m58.405306012s) [noise=true , address=145.79.227.8:4340 , piece_id=2XM5IYMXC3T4RZGP5STFD7X44GA5LLJ2YBLIZKZCMPV2EGMITP5Q, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.624481034 (505h11m58.495564069s) [noise=true , address=64.49.13.155:4924 , piece_id=5WEKSQI423JPQAZYOB27OV4RHWFVWSNH3A3XB3CLCS7T3DPRKPAQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.650754864 (505h11m58.521837909s) [noise=true , address=5.231.73.5:4029 , piece_id=HJICME4ZINBCPHVF2KUXILTQQIAYKQGIL5E4FT7WQHPX23YOEQ4A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.673199081 (505h11m58.544282117s) [noise=true , address=77.90.31.13:4646 , piece_id=BRJT6UUDGWS5T3KWTAW4ESJXFULND4FB6VQVYOSYRECDNNJ56SJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.745181288 (505h11m58.616264323s) [noise=true , address=77.90.31.12:4610 , piece_id=44TLRO3WIRAF45DAXO3TPQ2GHC2U3OFN2KLU73HG563552DGWUMA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:58.779227909 (505h11m58.650310954s) [noise=true , address=5.231.73.2:4027 , piece_id=Y6HUQVZS4RDZFGJCFHXHGY5GDBVNGZUNUAC5OXORLLD4GWA3DRLQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.073598071 (505h11m58.944681106s) [noise=true , address=77.90.31.17:4784 , piece_id=7CMDKQQHDP3P7M23U3ZP2GJVQC4KUCKCB4S6BBXHUNWBX6KS3DHQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.116295695 (505h11m58.98737874s) [noise=true , address=77.90.31.14:4671 , piece_id=6TIM4CCDRYOS2RIB4MJXG55NHN5AK5IM2WCJBRFZQTFGFBR7RUXQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.196854433 (505h11m59.067937478s) [noise=true , address=193.149.164.143:4703 , piece_id=EG5EHWBVYP4P3CCCDTO2IWZDNTQCMI7JTSRECYNMQVDBXQKN2WCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.426409626 (505h11m59.297492671s) [noise=true , address=145.79.227.44:4200 , piece_id=732NRQEMNGUE6WNMUFDRB3IEOOZKGLRPDLGOID54ADB4WHQW6QCQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535650219 (505h11m59.406733264s) [noise=true , address=145.79.227.33:4897 , piece_id=QLFZSNPKDCEFA5TQ5TCUIT3T7A4JQ3CX3AT5T4N7M64GEYZ647LA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: <nil> Dec 26 16:40:59.535675147 (505h11m59.406758192s) [seg_pos={0 0}, upload=443308]: Segment reached optimal threshold of 14 pieces. Dec 26 16:40:59.535893853 (505h11m59.406976898s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR): protocol: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34966->145.79.227.5:4255: use of closed network connection Dec 26 16:40:59.535902018 (505h11m59.406985053s) [noise=true , address=145.79.227.5:4255 , piece_id=2BXGXSZ76NKMPKLDE5NRJ622SFOLZT3FYAKOJ7EQSGKO2JBYDTJA, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536140953 (505h11m59.407224008s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536148949 (505h11m59.407231994s) [noise=true , address=145.79.227.40:4067 , piece_id=7ATX5TWB3NK4BARIMVPL4SYF2KUOCUR7JGYBCAK4W46EZASZIPSQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536341756 (505h11m59.407424791s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33344->77.90.31.21:4905: use of closed network connection Dec 26 16:40:59.536349891 (505h11m59.407432936s) [noise=true , address=77.90.31.21:4905 , piece_id=CARCPTBOEX4JRA3JHLGBMMSHYRZTFVVQALZ4WHBVJV7LOC5JCM5A, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536580220 (505h11m59.407663265s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128qNJJSSwMBPvN8wmTcUeMWAJxVmpjWjwA6WTKxBNCmT2FdFJn): protocol: expected piece hash; context canceled; EOF Dec 26 16:40:59.536592623 (505h11m59.407675658s) [noise=true , address=77.90.31.18:4812 , piece_id=4YFXYQYUW3FBVXMVJX47PLUU6QBJWXKSWULXTO35CHVZXPLVPQEQ, seg_pos={0 0}, upload=443308]: Long tail cancellation due to optimal threshold Dec 26 16:40:59.536637570 (505h11m59.407720605s) [seg_pos={0 0}, upload=443308]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 16:40:59.536765813 (505h11m59.407848858s) [upload=443308]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:40:59.536773297 (505h11m59.407856342s) [upload=443308]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:40:59.833034932 (505h11m59.704117977s) [upload=443308]: Done uploading segments... Dec 26 16:40:59.833058236 (505h11m59.704141271s) [upload=443308]: Upload finished. err: <nil> Duration: 33.596619228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 12.45127019s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:45:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 12.451301357s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [...........XXX...X...XX.....XX..........X......XXX...X......] duration: 5.000709032s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:44:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000733408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X...] duration: 6.108979301s trace: https://jaeger.dev.storj.io/trace/7c1ebe55b0448509 FAIL (2025-12-26 16:44:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA) context deadline exceeded ======LOGS====== ents... Dec 26 16:42:15.287484886 (505h13m15.158567931s) [upload=443326]: Got next segment. Inline: false Dec 26 16:42:15.287525874 (505h13m15.158608919s) [upload=443326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:42:15.287531806 (505h13m15.158614841s) [upload=443326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:42:15.539659110 (505h13m15.410742165s) [seg_pos={0 0}, upload=443326]: Begin upload segment... Dec 26 16:42:15.539714665 (505h13m15.41079771s) [seg_pos={0 0}, upload=443326]: Done begin upload segment. Dec 26 16:42:15.539742378 (505h13m15.410825413s) [noise=true , address=145.79.227.50:4374 , piece_id=DS7MCCAYIDWG6IBGLGU3GWJK4GZGDJJAH2ELP7CP4WUYR2KPPWZA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.539810518 (505h13m15.410893553s) [noise=true , address=145.79.227.12:4767 , piece_id=AMDHJU7TH4G2HVM2OSJ4OTSJJWBNPZKNCGEUJ3QHH232YIWQ5WFA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540298437 (505h13m15.411381482s) [noise=true , address=145.79.227.11:4732 , piece_id=YFPFZKKHILV4Q2674MJY4QFFJVJ6ILBDLAI5EINQQ5XFXUMHK3DQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540575104 (505h13m15.411658139s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540939098 (505h13m15.412022133s) [noise=true , address=145.79.227.48:4301 , piece_id=DDGMKU3SO3OG7N26A46CGRFIR6NDW5HQRQYVLBDIEAABH4PIFGKQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.541288553 (505h13m15.412371588s) [noise=true , address=145.79.227.32:4836 , piece_id=DSY3GJEJOP75QHYX6M655MSOXVYTLLPYB3AMVC7GAIGBM3LNCL2A, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.541665580 (505h13m15.412748615s) [noise=true , address=145.79.227.5:4222 , piece_id=C42KVZ2ZVNQJCU65NQDKLREJKCFJUUOV6X2KIGLDN75UBRK6JWTA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.542056155 (505h13m15.41313919s) [noise=true , address=145.79.227.47:4265 , piece_id=AVVNURIXLPZ33ZLYIR45L25PPBIHXP27B7SKKM7AYAMXQQ55QTNA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.542386443 (505h13m15.413469488s) [noise=true , address=45.91.116.5:4297 , piece_id=GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.543833599 (505h13m15.414916634s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.544814948 (505h13m15.415897993s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.546040822 (505h13m15.417123857s) [noise=true , address=145.79.227.15:4875 , piece_id=NODF22YJ2FI6VIZFOPMZF4NESXP3QN5PQU5NP7EIGNWP5A24CJNQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.547299118 (505h13m15.418382163s) [noise=true , address=145.79.227.52:4446 , piece_id=2OFNORXTUF3ZM5LSXMGU3QM2M4ICVVXZHSJK7EBOZGF7HTQZML3A, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.548767193 (505h13m15.419850238s) [noise=true , address=145.79.227.13:4803 , piece_id=ETOIFVMCRWWLU3CEHKROI6FK6GM7CI3CLYMSEBQDLV765MHYDJBA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.550118606 (505h13m15.421201641s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.551361323 (505h13m15.422444358s) [noise=true , address=145.79.227.42:4122 , piece_id=5DBWMUF6ZB6T4BTMWBWOL347GMEFH4KEUSV3ECKFYWMW6DQCWD6Q, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:17.259877936 (505h13m17.130960981s) [upload=443326]: Next returned nil segment Dec 26 16:42:17.259896352 (505h13m17.130979397s) [upload=443326]: Waiting for error group managing segments... Dec 26 16:42:17.539066701 (505h13m17.410149736s) [noise=true , address=145.79.227.32:4836 , piece_id=DSY3GJEJOP75QHYX6M655MSOXVYTLLPYB3AMVC7GAIGBM3LNCL2A, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.548192668 (505h13m17.419275713s) [noise=true , address=145.79.227.48:4301 , piece_id=DDGMKU3SO3OG7N26A46CGRFIR6NDW5HQRQYVLBDIEAABH4PIFGKQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.555516845 (505h13m17.42659989s) [noise=true , address=145.79.227.52:4446 , piece_id=2OFNORXTUF3ZM5LSXMGU3QM2M4ICVVXZHSJK7EBOZGF7HTQZML3A, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.558493903 (505h13m17.429576939s) [noise=true , address=145.79.227.47:4265 , piece_id=AVVNURIXLPZ33ZLYIR45L25PPBIHXP27B7SKKM7AYAMXQQ55QTNA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.563433349 (505h13m17.434516394s) [noise=true , address=145.79.227.11:4732 , piece_id=YFPFZKKHILV4Q2674MJY4QFFJVJ6ILBDLAI5EINQQ5XFXUMHK3DQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.568861816 (505h13m17.439944861s) [noise=true , address=145.79.227.15:4875 , piece_id=NODF22YJ2FI6VIZFOPMZF4NESXP3QN5PQU5NP7EIGNWP5A24CJNQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.572164695 (505h13m17.44324773s) [noise=true , address=145.79.227.50:4374 , piece_id=DS7MCCAYIDWG6IBGLGU3GWJK4GZGDJJAH2ELP7CP4WUYR2KPPWZA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.572447454 (505h13m17.443530499s) [noise=true , address=145.79.227.42:4122 , piece_id=5DBWMUF6ZB6T4BTMWBWOL347GMEFH4KEUSV3ECKFYWMW6DQCWD6Q, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.619080281 (505h13m17.490163326s) [noise=true , address=145.79.227.5:4222 , piece_id=C42KVZ2ZVNQJCU65NQDKLREJKCFJUUOV6X2KIGLDN75UBRK6JWTA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.630157514 (505h13m17.501240549s) [noise=true , address=145.79.227.13:4803 , piece_id=ETOIFVMCRWWLU3CEHKROI6FK6GM7CI3CLYMSEBQDLV765MHYDJBA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.769512389 (505h13m17.640595424s) [noise=true , address=145.79.227.12:4767 , piece_id=AMDHJU7TH4G2HVM2OSJ4OTSJJWBNPZKNCGEUJ3QHH232YIWQ5WFA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.093770894 (505h13m17.96485393s) [noise=true , address=45.91.116.5:4297 , piece_id=GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.109269015 (505h13m17.98035206s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.109288753 (505h13m17.980371798s) [seg_pos={0 0}, upload=443326]: Segment reached optimal threshold of 13 pieces. Dec 26 16:42:18.109480638 (505h13m17.980563683s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection Dec 26 16:42:18.109490316 (505h13m17.980573382s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109549650 (505h13m17.980632685s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection Dec 26 16:42:18.109556072 (505h13m17.980639107s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109739211 (505h13m17.980822266s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 16:42:18.109750192 (505h13m17.980833227s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109769067 (505h13m17.980852102s) [seg_pos={0 0}, upload=443326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:42:18.109965982 (505h13m17.981049028s) [upload=443326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:42:18.109975440 (505h13m17.981058505s) [upload=443326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:42:18.398418937 (505h13m18.269501982s) [upload=443326]: Done uploading segments... Dec 26 16:42:18.398441982 (505h13m18.269525027s) [upload=443326]: Upload finished. err: <nil> Duration: 6.108994841s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.903434609s trace: https://jaeger.dev.storj.io/trace/5ea564f725cf1502 FAIL (2025-12-26 16:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== upload=443078]: Uploading piece... Dec 26 16:25:11.740679859 (504h56m11.611762904s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.740794657 (504h56m11.611877702s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741129886 (504h56m11.612212921s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741530959 (504h56m11.612614004s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.741711542 (504h56m11.612794587s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742253645 (504h56m11.61333669s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.742829702 (504h56m11.613912747s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743358599 (504h56m11.614441644s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.743973339 (504h56m11.615056384s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:11.744744689 (504h56m11.615827734s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Uploading piece... Dec 26 16:25:12.570446138 (504h56m12.441529183s) [upload=443078]: Next returned nil segment Dec 26 16:25:12.570464414 (504h56m12.441547449s) [upload=443078]: Waiting for error group managing segments... Dec 26 16:25:12.964898154 (504h56m12.835981199s) [noise=true , address=185.167.204.206:4045 , piece_id=65X676FJP3VUQCTNVWZJMXQMWGO4KEUMWGGEEKAU5DYV6HOEWBRQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.964982896 (504h56m12.836065931s) [noise=true , address=185.167.204.205:4005 , piece_id=RFU376WR3GOSNZFDBJAVO66RUVZC3LPILOECNXYBAC7MMDXRYZZQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965621882 (504h56m12.836704927s) [noise=true , address=185.167.204.205:4007 , piece_id=3QIOKQKK5TOS4RQ2PJTFC2LF67N4JCZNAHTCNQB7IZU53MCY2K7Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.965905812 (504h56m12.836988868s) [noise=true , address=185.167.204.205:4024 , piece_id=CR62L37HSF7NLRYP33CJJMRH3MT7BL6LAVNDME6XLYJI42GXGQWQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975145387 (504h56m12.846228422s) [noise=true , address=185.167.204.206:4069 , piece_id=MQ3ELJHF4TPHBNSXG6AO7Z73ELKHUPA2S3JSAMB5XSPHQTPXFQLQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.975614930 (504h56m12.846697965s) [noise=true , address=185.167.204.205:4025 , piece_id=WSQ4D4ABXOXAMUVP2XC57USHBKKAN3BOXXJL5ZTA6SWZPJ4DQ4NQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976130462 (504h56m12.847213507s) [noise=true , address=185.167.204.205:4023 , piece_id=BV2BKYUBJSOZCJP7BB45CW4IEYGYRTGPVBO5K5PRSNMRTMEFECAQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:12.976669368 (504h56m12.847752413s) [noise=true , address=185.167.204.206:4071 , piece_id=MWRRVPDOO6ZBD6PE7O3PN2PFP633SXXQK5BAHD5TUL6WLFG4MRVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.683093112 (504h56m13.554176157s) [noise=true , address=81.171.14.70:4030 , piece_id=IVDCI2VXEOQMXEGX2WG7BXXZYY3YDN2HFQNV5GZM34DLJJN6P3BA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.700306490 (504h56m13.571389535s) [noise=true , address=81.171.14.70:4034 , piece_id=TNOFONTDEX7Q6YGHHAX7KQRIVR2ILE46XU4PZYD6UGCQECUVYFIA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.758935049 (504h56m13.630018094s) [noise=true , address=81.171.14.70:4032 , piece_id=5C3JEAU2QCDKLJWPMCB66MC7B5NPEDERO7DCZBON3YYJB4ZCRNSA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775430309 (504h56m13.646513344s) [noise=true , address=81.171.14.70:4042 , piece_id=5L4R6BXN6UBRHY7DODHKY5CDDEOW3SHWTTIEUD566WA5H7BVSV4A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:13.775933507 (504h56m13.647016542s) [noise=true , address=81.171.14.70:4031 , piece_id=3QO4VJ2ZQ4VEYEHPGNWDYTMG7RPEHUJ3IWOBWULIWDHCPDXNT3VA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.195414712 (504h56m14.066497747s) [noise=true , address=89.149.201.147:4002 , piece_id=2TFPPT6AUDZARDTIDI666GPNKVGBLQAUT7TE4O3PLCK4Z5E36WIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.285943735 (504h56m14.15702677s) [noise=true , address=89.149.201.147:4024 , piece_id=4S6V6MCIVJCRR3PFZD23HXTJ42LD3FNQUVPWUBVCBYL6QUL2HQTA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.286477421 (504h56m14.157560466s) [noise=true , address=81.171.14.70:4028 , piece_id=OB3JEBFMXNLOYHCXGUXGXAWQZH44CQ4LU5ZIDHEZQYEZQYJ2EDVQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.294503615 (504h56m14.16558665s) [noise=true , address=89.149.201.147:4023 , piece_id=B7BGNDD4U7EVRXNOJOGRIXHIIZFWULBC54LYKIGJ7L5N2U7ZLXVA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.331128254 (504h56m14.202211299s) [noise=true , address=89.149.201.147:4006 , piece_id=CW37U4D37UZ75KNRHOKQZWNSUFC67J43VOBEWAVPJ6ZRCUXMIJ3A, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.332130333 (504h56m14.203213368s) [noise=true , address=81.171.14.70:4038 , piece_id=P4CIJ2MW6GKZEZ2LH4SSDNZRQYAHVTDVZ753N4ML4ZCDFZK7S44Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.335360984 (504h56m14.20644407s) [noise=true , address=89.149.201.147:4020 , piece_id=2YX2RWUDFOKCCJNEDWXMGVCZ7MANAUB6I5FYXGG5CS5LINXAZ7AQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.400928806 (504h56m14.272011841s) [noise=true , address=89.149.201.147:4012 , piece_id=TPVJ44WWDE6SB7VIBY55E6Y3EEZRI6AN3SCSCVOMFIDS3QON22PA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.411019842 (504h56m14.282102887s) [noise=true , address=81.171.14.70:4045 , piece_id=GNL2KKBSGFM5W7QSS364AN3G4EF54XPIFMTT5GFTTJLTH7W5KKAA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.418301518 (504h56m14.289384553s) [noise=true , address=89.149.201.147:4010 , piece_id=2HWUTORHG5NNNDM4DFOCUCNIFPQ7Z5N6MP3QT3YYOHQXNCIOFOXQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.441258031 (504h56m14.312341066s) [noise=true , address=81.171.14.70:4037 , piece_id=ZFWYVO5BWFDDNTG7P56GEALCYLKDM2UJ7RZRDQ2HUMPUV3N4F4QQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:14.462688989 (504h56m14.333772024s) [noise=true , address=89.149.201.147:4018 , piece_id=K3IIKQUY65TFZGXZ3UG73G3Y2TAKD2C6ACGKLCER4VB5BVNPFAIQ, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109832182 (504h56m14.980915217s) [noise=true , address=89.149.201.147:4015 , piece_id=LLMVGCN2ANGGI7DFSJJFKBFEEKR7ZYNMVS2GB5L7LPKWQAE2ZRBA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: <nil> Dec 26 16:25:15.109853373 (504h56m14.980936408s) [seg_pos={0 0}, upload=443078]: Segment reached optimal threshold of 26 pieces. Dec 26 16:25:15.110009039 (504h56m14.981092074s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sbZWJFMVsppCpbFVfn33bx5uGkhtFpzJm8a2i8DoAfQNsWY5F): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110014831 (504h56m14.981097876s) [noise=true , address=89.149.201.147:4009 , piece_id=A2H664CG5GRITTNES3TZ6SU4DW3PTWJTOH5NJC3C2F5P5GETSO3Q, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110234038 (504h56m14.981317073s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12o3f1bhpf5JrqLvdu57xT2JFxNwAzkUT4bW8qdjn5nbJ1mQDko): protocol: expected piece hash; context canceled; EOF Dec 26 16:25:15.110252523 (504h56m14.981335568s) [noise=true , address=81.171.14.70:4043 , piece_id=2XOGYYVWR6JHNTEZFIGPEALITCZTDXPHHVSKWN3D2EOQLOU7CABA, seg_pos={0 0}, upload=443078]: Long tail cancellation due to optimal threshold Dec 26 16:25:15.110279705 (504h56m14.98136274s) [seg_pos={0 0}, upload=443078]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 26 16:25:15.110393751 (504h56m14.981476787s) [upload=443078]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:25:15.110400916 (504h56m14.981483961s) [upload=443078]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:25:15.347796749 (504h56m15.218879794s) [upload=443078]: Done uploading segments... Dec 26 16:25:15.347816526 (504h56m15.218899571s) [upload=443078]: Upload finished. err: <nil> Duration: 7.903476427s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [............XXX...X...XX.....XX..........X......XXX...X.....] duration: 4.66707745s trace: https://jaeger.dev.storj.io/trace/69cd2adc1b6c0e6d FAIL (2025-12-26 16:44:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.667106623s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.....X..XX..XXX.X...XX.X.XX.X....XX...X...XXX..X..X...X..] duration: 6.305511914s trace: https://jaeger.dev.storj.io/trace/7c1ebe55b0448509 FAIL (2025-12-26 16:44:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA) context deadline exceeded ======LOGS====== ents... Dec 26 16:42:15.287484886 (505h13m15.158567931s) [upload=443326]: Got next segment. Inline: false Dec 26 16:42:15.287525874 (505h13m15.158608919s) [upload=443326]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 16:42:15.287531806 (505h13m15.158614841s) [upload=443326]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 16:42:15.539659110 (505h13m15.410742165s) [seg_pos={0 0}, upload=443326]: Begin upload segment... Dec 26 16:42:15.539714665 (505h13m15.41079771s) [seg_pos={0 0}, upload=443326]: Done begin upload segment. Dec 26 16:42:15.539742378 (505h13m15.410825413s) [noise=true , address=145.79.227.50:4374 , piece_id=DS7MCCAYIDWG6IBGLGU3GWJK4GZGDJJAH2ELP7CP4WUYR2KPPWZA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.539810518 (505h13m15.410893553s) [noise=true , address=145.79.227.12:4767 , piece_id=AMDHJU7TH4G2HVM2OSJ4OTSJJWBNPZKNCGEUJ3QHH232YIWQ5WFA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540298437 (505h13m15.411381482s) [noise=true , address=145.79.227.11:4732 , piece_id=YFPFZKKHILV4Q2674MJY4QFFJVJ6ILBDLAI5EINQQ5XFXUMHK3DQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540575104 (505h13m15.411658139s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.540939098 (505h13m15.412022133s) [noise=true , address=145.79.227.48:4301 , piece_id=DDGMKU3SO3OG7N26A46CGRFIR6NDW5HQRQYVLBDIEAABH4PIFGKQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.541288553 (505h13m15.412371588s) [noise=true , address=145.79.227.32:4836 , piece_id=DSY3GJEJOP75QHYX6M655MSOXVYTLLPYB3AMVC7GAIGBM3LNCL2A, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.541665580 (505h13m15.412748615s) [noise=true , address=145.79.227.5:4222 , piece_id=C42KVZ2ZVNQJCU65NQDKLREJKCFJUUOV6X2KIGLDN75UBRK6JWTA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.542056155 (505h13m15.41313919s) [noise=true , address=145.79.227.47:4265 , piece_id=AVVNURIXLPZ33ZLYIR45L25PPBIHXP27B7SKKM7AYAMXQQ55QTNA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.542386443 (505h13m15.413469488s) [noise=true , address=45.91.116.5:4297 , piece_id=GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.543833599 (505h13m15.414916634s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.544814948 (505h13m15.415897993s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.546040822 (505h13m15.417123857s) [noise=true , address=145.79.227.15:4875 , piece_id=NODF22YJ2FI6VIZFOPMZF4NESXP3QN5PQU5NP7EIGNWP5A24CJNQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.547299118 (505h13m15.418382163s) [noise=true , address=145.79.227.52:4446 , piece_id=2OFNORXTUF3ZM5LSXMGU3QM2M4ICVVXZHSJK7EBOZGF7HTQZML3A, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.548767193 (505h13m15.419850238s) [noise=true , address=145.79.227.13:4803 , piece_id=ETOIFVMCRWWLU3CEHKROI6FK6GM7CI3CLYMSEBQDLV765MHYDJBA, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.550118606 (505h13m15.421201641s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:15.551361323 (505h13m15.422444358s) [noise=true , address=145.79.227.42:4122 , piece_id=5DBWMUF6ZB6T4BTMWBWOL347GMEFH4KEUSV3ECKFYWMW6DQCWD6Q, seg_pos={0 0}, upload=443326]: Uploading piece... Dec 26 16:42:17.259877936 (505h13m17.130960981s) [upload=443326]: Next returned nil segment Dec 26 16:42:17.259896352 (505h13m17.130979397s) [upload=443326]: Waiting for error group managing segments... Dec 26 16:42:17.539066701 (505h13m17.410149736s) [noise=true , address=145.79.227.32:4836 , piece_id=DSY3GJEJOP75QHYX6M655MSOXVYTLLPYB3AMVC7GAIGBM3LNCL2A, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.548192668 (505h13m17.419275713s) [noise=true , address=145.79.227.48:4301 , piece_id=DDGMKU3SO3OG7N26A46CGRFIR6NDW5HQRQYVLBDIEAABH4PIFGKQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.555516845 (505h13m17.42659989s) [noise=true , address=145.79.227.52:4446 , piece_id=2OFNORXTUF3ZM5LSXMGU3QM2M4ICVVXZHSJK7EBOZGF7HTQZML3A, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.558493903 (505h13m17.429576939s) [noise=true , address=145.79.227.47:4265 , piece_id=AVVNURIXLPZ33ZLYIR45L25PPBIHXP27B7SKKM7AYAMXQQ55QTNA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.563433349 (505h13m17.434516394s) [noise=true , address=145.79.227.11:4732 , piece_id=YFPFZKKHILV4Q2674MJY4QFFJVJ6ILBDLAI5EINQQ5XFXUMHK3DQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.568861816 (505h13m17.439944861s) [noise=true , address=145.79.227.15:4875 , piece_id=NODF22YJ2FI6VIZFOPMZF4NESXP3QN5PQU5NP7EIGNWP5A24CJNQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.572164695 (505h13m17.44324773s) [noise=true , address=145.79.227.50:4374 , piece_id=DS7MCCAYIDWG6IBGLGU3GWJK4GZGDJJAH2ELP7CP4WUYR2KPPWZA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.572447454 (505h13m17.443530499s) [noise=true , address=145.79.227.42:4122 , piece_id=5DBWMUF6ZB6T4BTMWBWOL347GMEFH4KEUSV3ECKFYWMW6DQCWD6Q, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.619080281 (505h13m17.490163326s) [noise=true , address=145.79.227.5:4222 , piece_id=C42KVZ2ZVNQJCU65NQDKLREJKCFJUUOV6X2KIGLDN75UBRK6JWTA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.630157514 (505h13m17.501240549s) [noise=true , address=145.79.227.13:4803 , piece_id=ETOIFVMCRWWLU3CEHKROI6FK6GM7CI3CLYMSEBQDLV765MHYDJBA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:17.769512389 (505h13m17.640595424s) [noise=true , address=145.79.227.12:4767 , piece_id=AMDHJU7TH4G2HVM2OSJ4OTSJJWBNPZKNCGEUJ3QHH232YIWQ5WFA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.093770894 (505h13m17.96485393s) [noise=true , address=45.91.116.5:4297 , piece_id=GXJMYIVD4PWIRAZEHIELQVFCSO5BC4ZTVSC4NXW3ELBMGZCOJIDA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.109269015 (505h13m17.98035206s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHLJZL553VYDC76SRKSAL5IKSVMOV6BZOH3KCW2VWC7UE24H7HJQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: <nil> Dec 26 16:42:18.109288753 (505h13m17.980371798s) [seg_pos={0 0}, upload=443326]: Segment reached optimal threshold of 13 pieces. Dec 26 16:42:18.109480638 (505h13m17.980563683s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37840->45.91.116.13:4154: use of closed network connection Dec 26 16:42:18.109490316 (505h13m17.980573382s) [noise=true , address=45.91.116.13:4154 , piece_id=2NGV2JXC7MIXF3M26TO4URHR4LX4CBWJLETY5FDFYCJWNGMAAKBQ, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109549650 (505h13m17.980632685s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48030->45.91.116.12:4227: use of closed network connection Dec 26 16:42:18.109556072 (505h13m17.980639107s) [noise=true , address=45.91.116.12:4227 , piece_id=MV2DAMZE6FY5NR27HPKNGETBZEN34LEQSOOUIT5N6OTXXGBXZQOA, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109739211 (505h13m17.980822266s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 16:42:18.109750192 (505h13m17.980833227s) [noise=true , address=145.79.227.40:4051 , piece_id=N7XR6BGZI7VYKAEATP24L2I3HNQKG3FOX772MX6GMIOMVY4ZC7PA, seg_pos={0 0}, upload=443326]: Long tail cancellation due to optimal threshold Dec 26 16:42:18.109769067 (505h13m17.980852102s) [seg_pos={0 0}, upload=443326]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 16:42:18.109965982 (505h13m17.981049028s) [upload=443326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 16:42:18.109975440 (505h13m17.981058505s) [upload=443326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 16:42:18.398418937 (505h13m18.269501982s) [upload=443326]: Done uploading segments... Dec 26 16:42:18.398441982 (505h13m18.269525027s) [upload=443326]: Upload finished. err: <nil> Duration: 6.305549636s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)