Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X...............XX..........................X.X] duration: 30.003896876s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 30.003918438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...........X...............XX..........................X.] duration: 6.125205265s trace: https://jaeger.dev.storj.io/trace/38cbe59e5e3bb35c FAIL (2025-12-25 10:08:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A) context deadline exceeded ======LOGS====== (474h38m5.704587591s) [upload=416618]: Uploading segments... Dec 25 10:07:05.833742549 (474h38m5.704825584s) [upload=416618]: Got next segment. Inline: false Dec 25 10:07:05.834408427 (474h38m5.705491482s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:07:05.834417854 (474h38m5.705500889s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:07:06.081355973 (474h38m5.952439018s) [seg_pos={0 0}, upload=416618]: Begin upload segment... Dec 25 10:07:06.081422761 (474h38m5.952505806s) [seg_pos={0 0}, upload=416618]: Done begin upload segment. Dec 25 10:07:06.081473036 (474h38m5.952556081s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.081510076 (474h38m5.952593121s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082120720 (474h38m5.953203755s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082511433 (474h38m5.953594468s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083013499 (474h38m5.954096544s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083436375 (474h38m5.95451941s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083811178 (474h38m5.954894223s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084218294 (474h38m5.955301339s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084612624 (474h38m5.955695669s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086269009 (474h38m5.957352054s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086516650 (474h38m5.957599695s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.088408151 (474h38m5.959491186s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.090281689 (474h38m5.961364734s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.091837741 (474h38m5.962920776s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.093299495 (474h38m5.96438254s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.094836873 (474h38m5.965919908s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:07.807999056 (474h38m7.679082091s) [upload=416618]: Next returned nil segment Dec 25 10:07:07.808035075 (474h38m7.67911811s) [upload=416618]: Waiting for error group managing segments... Dec 25 10:07:08.095233103 (474h38m7.966316148s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.101219963 (474h38m7.972302998s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103194153 (474h38m7.974277198s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103253335 (474h38m7.97433637s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.113152104 (474h38m7.984235149s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.116079047 (474h38m7.987162093s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.120584797 (474h38m7.991667832s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.125144830 (474h38m7.996227875s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.128055221 (474h38m7.999138256s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.321456498 (474h38m8.192539533s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.372613558 (474h38m8.243696593s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.383542358 (474h38m8.254625393s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449614751 (474h38m8.320697786s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449638396 (474h38m8.320721431s) [seg_pos={0 0}, upload=416618]: Segment reached optimal threshold of 13 pieces. Dec 25 10:07:08.450036203 (474h38m8.321119268s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection Dec 25 10:07:08.450047766 (474h38m8.321130811s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450101538 (474h38m8.321184583s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 10:07:08.450125233 (474h38m8.321208278s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450326667 (474h38m8.321409712s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection Dec 25 10:07:08.450335664 (474h38m8.321418699s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450356743 (474h38m8.321439798s) [seg_pos={0 0}, upload=416618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:07:08.450457124 (474h38m8.321540159s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:07:08.450462344 (474h38m8.321545389s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:07:08.741746594 (474h38m8.612829669s) [upload=416618]: Done uploading segments... Dec 25 10:07:08.741774858 (474h38m8.612857893s) [upload=416618]: Upload finished. err: <nil> Duration: 6.125245291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...........X...............XX..........................X] duration: 30.002440363s trace: https://jaeger.dev.storj.io/trace/38cbe59e5e3bb35c FAIL (2025-12-25 10:07:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A) context deadline exceeded ======LOGS====== (474h38m5.704587591s) [upload=416618]: Uploading segments... Dec 25 10:07:05.833742549 (474h38m5.704825584s) [upload=416618]: Got next segment. Inline: false Dec 25 10:07:05.834408427 (474h38m5.705491482s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:07:05.834417854 (474h38m5.705500889s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:07:06.081355973 (474h38m5.952439018s) [seg_pos={0 0}, upload=416618]: Begin upload segment... Dec 25 10:07:06.081422761 (474h38m5.952505806s) [seg_pos={0 0}, upload=416618]: Done begin upload segment. Dec 25 10:07:06.081473036 (474h38m5.952556081s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.081510076 (474h38m5.952593121s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082120720 (474h38m5.953203755s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082511433 (474h38m5.953594468s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083013499 (474h38m5.954096544s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083436375 (474h38m5.95451941s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083811178 (474h38m5.954894223s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084218294 (474h38m5.955301339s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084612624 (474h38m5.955695669s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086269009 (474h38m5.957352054s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086516650 (474h38m5.957599695s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.088408151 (474h38m5.959491186s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.090281689 (474h38m5.961364734s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.091837741 (474h38m5.962920776s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.093299495 (474h38m5.96438254s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.094836873 (474h38m5.965919908s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:07.807999056 (474h38m7.679082091s) [upload=416618]: Next returned nil segment Dec 25 10:07:07.808035075 (474h38m7.67911811s) [upload=416618]: Waiting for error group managing segments... Dec 25 10:07:08.095233103 (474h38m7.966316148s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.101219963 (474h38m7.972302998s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103194153 (474h38m7.974277198s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103253335 (474h38m7.97433637s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.113152104 (474h38m7.984235149s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.116079047 (474h38m7.987162093s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.120584797 (474h38m7.991667832s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.125144830 (474h38m7.996227875s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.128055221 (474h38m7.999138256s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.321456498 (474h38m8.192539533s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.372613558 (474h38m8.243696593s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.383542358 (474h38m8.254625393s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449614751 (474h38m8.320697786s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449638396 (474h38m8.320721431s) [seg_pos={0 0}, upload=416618]: Segment reached optimal threshold of 13 pieces. Dec 25 10:07:08.450036203 (474h38m8.321119268s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection Dec 25 10:07:08.450047766 (474h38m8.321130811s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450101538 (474h38m8.321184583s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 10:07:08.450125233 (474h38m8.321208278s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450326667 (474h38m8.321409712s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection Dec 25 10:07:08.450335664 (474h38m8.321418699s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450356743 (474h38m8.321439798s) [seg_pos={0 0}, upload=416618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:07:08.450457124 (474h38m8.321540159s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:07:08.450462344 (474h38m8.321545389s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:07:08.741746594 (474h38m8.612829669s) [upload=416618]: Done uploading segments... Dec 25 10:07:08.741774858 (474h38m8.612857893s) [upload=416618]: Upload finished. err: <nil> Duration: 30.002462325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...........X...............XX..........................] duration: 5.978138819s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:06:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.978163688s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.684280993s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:06:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684314146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........X...............XX.........................] duration: 5.711735785s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.711812021s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.580609147s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:05:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580634245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.590713917s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:05:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590745786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........X...............XX........................] duration: 5.824581235s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:05:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.824606293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.728241984s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:04:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.728295577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.613290958s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:04:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.61332353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X...............XX.......................] duration: 5.843032332s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.843064082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...........X...............XX......................] duration: 5.711445217s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:03:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.711471546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.588832393s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:02:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588859835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........X...............XX.....................] duration: 5.678770961s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:02:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.678798764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.730454006s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.730479905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.025367151s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025399814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.........X...........X...............XX....................] duration: 5.99135213s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.991546921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.415345567s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:00:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415400711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.........X...........X...............XX...................] duration: 6.011706447s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.011731505s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.865385278s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:59:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.865408763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.........X...........X...............XX..................] duration: 5.765173446s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.765221669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.........X...........X...............XX.................] duration: 5.923426475s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.923450791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.424342998s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424368097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.405510427s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40565302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.........X...........X...............XX................] duration: 6.242879239s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.2429111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 4.470727863s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.47075268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.430758276s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430785548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.........X...........X...............XX...............] duration: 5.672214878s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:56:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.672241218s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 3.454943376s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:56:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454978433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.........X...........X...............XX..............] duration: 5.763534214s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.763560525s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 3.490428227s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:55:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490453264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.........X...........X...............XX.............] duration: 5.687600855s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.687774215s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.........X...........X...............XX............] duration: 5.906169747s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.906198383s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.373423078s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:54:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.37345019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.........X...........X...............XX...........] duration: 5.921094206s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:53:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.921124895s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.59110866s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:53:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.591133727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.445774528s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:53:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.445798764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.........X...........X...............XX..........] duration: 5.953528517s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:52:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.953553443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.........X...........X...............XX.........] duration: 5.628252406s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:52:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.628279897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.621223102s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:51:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621243131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.815504527s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:51:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.815535215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.........X...........X...............XX........] duration: 5.661163774s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:51:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.661194693s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.515634654s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:50:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515667728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.........X...........X...............XX.......] duration: 5.680451643s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:50:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.680505487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.658334268s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:49:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.658362001s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.........X...........X...............XX......] duration: 5.681987678s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:49:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.682012825s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.594864208s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:48:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59488005s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.........X...........X...............XX.....] duration: 6.203633469s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.203662585s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.646646312s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:48:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.646672612s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)