Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.380707391s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:12:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.380737058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..XX.XXX...X.XX..XXX...............X....................XX.X] duration: 30.002307163s trace: https://jaeger.dev.storj.io/trace/4c4fb56c9197a891 FAIL (2025-12-26 21:11:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6VAQGQDLNXCETJFKPS6H2JVBFOIWGTLGUBEOTZBJ6ACY6UYRLOXQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: HSITSF4QNNQFNTAHA3ALFZFCZAFKVEPPE464UJZH4RL25PMG44YA) context deadline exceeded ======LOGS====== 6 21:11:11.245143283 (509h42m11.116226328s) [upload=447272]: Starting upload Dec 26 21:11:11.245219308 (509h42m11.116302353s) [upload=447272]: Done starting upload Dec 26 21:11:11.245328417 (509h42m11.116411462s) [upload=447272]: Uploading segments... Dec 26 21:11:11.245535852 (509h42m11.116618887s) [upload=447272]: Got next segment. Inline: false Dec 26 21:11:11.245604242 (509h42m11.116687287s) [upload=447272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:11:11.245611485 (509h42m11.116694531s) [upload=447272]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:11:11.501221728 (509h42m11.372304773s) [seg_pos={0 0}, upload=447272]: Begin upload segment... Dec 26 21:11:11.501265130 (509h42m11.372348165s) [seg_pos={0 0}, upload=447272]: Done begin upload segment. Dec 26 21:11:11.501278546 (509h42m11.372361591s) [noise=true , address=77.90.31.18:4808 , piece_id=6VAQGQDLNXCETJFKPS6H2JVBFOIWGTLGUBEOTZBJ6ACY6UYRLOXQ, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.501330034 (509h42m11.372413079s) [noise=true , address=45.91.116.5:4300 , piece_id=CYIA7VH5EIE565OI2FBS2H6IPTVTLEQLLJMMH2BELXKMO3HVEHFA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.501850775 (509h42m11.37293381s) [noise=true , address=145.79.227.31:4798 , piece_id=XSK7SPQIBPGSEALJFWLVUPDRVG2QRSMIPMIYIR7YF5KH5R4FA3SQ, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.502248462 (509h42m11.373331497s) [noise=true , address=45.91.116.6:4974 , piece_id=HSITSF4QNNQFNTAHA3ALFZFCZAFKVEPPE464UJZH4RL25PMG44YA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.502571959 (509h42m11.373655004s) [noise=true , address=145.79.227.52:4446 , piece_id=5UDTIDYPD5RKCL2B7Z3NJ2RYZBG4PH2SWKIBMRXU47ABF7INEWMQ, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.502896617 (509h42m11.373979652s) [noise=true , address=145.79.227.5:4222 , piece_id=IC5AYN2IHMBNKESR3B3QJTA3P7AUNK3PI7E3XA4UMPBZ6234GAIQ, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.503323479 (509h42m11.374406524s) [noise=true , address=145.79.227.48:4301 , piece_id=5HJ4R5Q7J7BXGER6PMVFEYYXN5WFXA2RTHNKQML4YPWE7JOL6CEQ, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.504111721 (509h42m11.375194756s) [noise=true , address=145.79.227.39:4013 , piece_id=OYZVNYZEBIQS52EB6UE2TWQFKG6LJLTGQJI6N3QAVJVHDBVIXNXA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.504599850 (509h42m11.375682885s) [noise=true , address=145.79.227.50:4374 , piece_id=IUETZVHURRFQDJZQYZC4YZXDJUR2JAZEWLRGRPHF2XFWFLAGNZPA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.504933586 (509h42m11.376016621s) [noise=true , address=77.90.31.10:4520 , piece_id=PTRH735CHZSTMUQ6HZWTF7PN4NA2KI3QCDCRK3TPXGOOYB7YSZ3Q, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.505230080 (509h42m11.376313115s) [noise=true , address=145.79.227.3:4151 , piece_id=MWHX7JZ7H4ZNBREYK4CM7HXDRM42JWXFCI6X644KBW3ADI6VNGOA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.505555449 (509h42m11.376638494s) [noise=true , address=145.79.227.42:4121 , piece_id=VNFA6CNJADQ2IDY6QLKQKCVNEJEHFJDGGHQQCB2IIJEPEKOYCNZA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.505979778 (509h42m11.377062813s) [noise=true , address=145.79.227.45:4231 , piece_id=ULG5YTBYB7U5XQIQC2MZURS5UUQJVGORI34VYXLJGC4ECDMZOQ2Q, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.506392975 (509h42m11.37747602s) [noise=true , address=145.79.227.47:4267 , piece_id=BM3SDBRLGHDFUCWAH7IQRAZIJKN36MBW2LJU6J5SDZ4QIYZEZ76Q, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.506653250 (509h42m11.377736285s) [noise=true , address=145.79.227.36:4519 , piece_id=3TT5RPYCBN4DUNV5END76QQYZNVF2A2UOHXUPNQOIRTRYHMKUAGA, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:11.506990933 (509h42m11.378073979s) [noise=true , address=45.91.116.12:4227 , piece_id=Q2HDFGH2QTFDJWRSXEJJ6XMVPQEVWLOAI7SVYDXEAFET4QKEOKCQ, seg_pos={0 0}, upload=447272]: Uploading piece... Dec 26 21:11:13.210328154 (509h42m13.081411189s) [upload=447272]: Next returned nil segment Dec 26 21:11:13.210344124 (509h42m13.081427159s) [upload=447272]: Waiting for error group managing segments... Dec 26 21:11:13.488644813 (509h42m13.359727848s) [noise=true , address=145.79.227.48:4301 , piece_id=5HJ4R5Q7J7BXGER6PMVFEYYXN5WFXA2RTHNKQML4YPWE7JOL6CEQ, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.511537706 (509h42m13.382620751s) [noise=true , address=145.79.227.50:4374 , piece_id=IUETZVHURRFQDJZQYZC4YZXDJUR2JAZEWLRGRPHF2XFWFLAGNZPA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.515474553 (509h42m13.386557588s) [noise=true , address=145.79.227.45:4231 , piece_id=ULG5YTBYB7U5XQIQC2MZURS5UUQJVGORI34VYXLJGC4ECDMZOQ2Q, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.517631770 (509h42m13.388714805s) [noise=true , address=145.79.227.52:4446 , piece_id=5UDTIDYPD5RKCL2B7Z3NJ2RYZBG4PH2SWKIBMRXU47ABF7INEWMQ, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.521647638 (509h42m13.392730673s) [noise=true , address=145.79.227.3:4151 , piece_id=MWHX7JZ7H4ZNBREYK4CM7HXDRM42JWXFCI6X644KBW3ADI6VNGOA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.524938273 (509h42m13.396021318s) [noise=true , address=145.79.227.36:4519 , piece_id=3TT5RPYCBN4DUNV5END76QQYZNVF2A2UOHXUPNQOIRTRYHMKUAGA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.531590350 (509h42m13.402673385s) [noise=true , address=145.79.227.31:4798 , piece_id=XSK7SPQIBPGSEALJFWLVUPDRVG2QRSMIPMIYIR7YF5KH5R4FA3SQ, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.539351800 (509h42m13.410434845s) [noise=true , address=145.79.227.47:4267 , piece_id=BM3SDBRLGHDFUCWAH7IQRAZIJKN36MBW2LJU6J5SDZ4QIYZEZ76Q, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.671069456 (509h42m13.542152492s) [noise=true , address=77.90.31.18:4808 , piece_id=6VAQGQDLNXCETJFKPS6H2JVBFOIWGTLGUBEOTZBJ6ACY6UYRLOXQ, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.743699027 (509h42m13.614782062s) [noise=true , address=77.90.31.10:4520 , piece_id=PTRH735CHZSTMUQ6HZWTF7PN4NA2KI3QCDCRK3TPXGOOYB7YSZ3Q, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:13.817923716 (509h42m13.689006761s) [noise=true , address=45.91.116.6:4974 , piece_id=HSITSF4QNNQFNTAHA3ALFZFCZAFKVEPPE464UJZH4RL25PMG44YA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:14.058392887 (509h42m13.929475922s) [noise=true , address=45.91.116.12:4227 , piece_id=Q2HDFGH2QTFDJWRSXEJJ6XMVPQEVWLOAI7SVYDXEAFET4QKEOKCQ, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:14.436078515 (509h42m14.30716155s) [noise=true , address=45.91.116.5:4300 , piece_id=CYIA7VH5EIE565OI2FBS2H6IPTVTLEQLLJMMH2BELXKMO3HVEHFA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: <nil> Dec 26 21:11:14.436104885 (509h42m14.30718793s) [seg_pos={0 0}, upload=447272]: Segment reached optimal threshold of 13 pieces. Dec 26 21:11:14.436299816 (509h42m14.307382861s) [noise=true , address=145.79.227.42:4121 , piece_id=VNFA6CNJADQ2IDY6QLKQKCVNEJEHFJDGGHQQCB2IIJEPEKOYCNZA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF Dec 26 21:11:14.436306618 (509h42m14.307389673s) [noise=true , address=145.79.227.42:4121 , piece_id=VNFA6CNJADQ2IDY6QLKQKCVNEJEHFJDGGHQQCB2IIJEPEKOYCNZA, seg_pos={0 0}, upload=447272]: Long tail cancellation due to optimal threshold Dec 26 21:11:14.436407410 (509h42m14.307490445s) [noise=true , address=145.79.227.5:4222 , piece_id=IC5AYN2IHMBNKESR3B3QJTA3P7AUNK3PI7E3XA4UMPBZ6234GAIQ, seg_pos={0 0}, upload=447272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:36210->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:36210->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36210->145.79.227.5:4222: use of closed network connection Dec 26 21:11:14.436427569 (509h42m14.307510614s) [noise=true , address=145.79.227.5:4222 , piece_id=IC5AYN2IHMBNKESR3B3QJTA3P7AUNK3PI7E3XA4UMPBZ6234GAIQ, seg_pos={0 0}, upload=447272]: Long tail cancellation due to optimal threshold Dec 26 21:11:14.436512110 (509h42m14.307595145s) [noise=true , address=145.79.227.39:4013 , piece_id=OYZVNYZEBIQS52EB6UE2TWQFKG6LJLTGQJI6N3QAVJVHDBVIXNXA, seg_pos={0 0}, upload=447272]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 26 21:11:14.436518903 (509h42m14.307601938s) [noise=true , address=145.79.227.39:4013 , piece_id=OYZVNYZEBIQS52EB6UE2TWQFKG6LJLTGQJI6N3QAVJVHDBVIXNXA, seg_pos={0 0}, upload=447272]: Long tail cancellation due to optimal threshold Dec 26 21:11:14.436595359 (509h42m14.307678394s) [seg_pos={0 0}, upload=447272]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:11:14.436697103 (509h42m14.307780138s) [upload=447272]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:11:14.436703565 (509h42m14.30778661s) [upload=447272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:11:14.741693716 (509h42m14.612776771s) [upload=447272]: Done uploading segments... Dec 26 21:11:14.741720848 (509h42m14.612803893s) [upload=447272]: Upload finished. err: <nil> Duration: 30.002317804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 27.280288597s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:11:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 27.280309708s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XX.XXX...X.XX..XXX...............X....................XX.] duration: 27.038198482s trace: https://jaeger.dev.storj.io/trace/412ab68d06408eac FAIL (2025-12-26 21:11:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 21:10:13.756848981 (509h41m13.627932016s) [upload=447260]: Starting upload Dec 26 21:10:13.756907482 (509h41m13.627990517s) [upload=447260]: Done starting upload Dec 26 21:10:13.757014996 (509h41m13.628098031s) [upload=447260]: Uploading segments... Dec 26 21:10:13.757094799 (509h41m13.628177834s) [upload=447260]: Got next segment. Inline: false Dec 26 21:10:13.757133873 (509h41m13.628216918s) [upload=447260]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:10:13.757140265 (509h41m13.62822331s) [upload=447260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:10:14.009399737 (509h41m13.880482792s) [seg_pos={0 0}, upload=447260]: Begin upload segment... Dec 26 21:10:14.009450544 (509h41m13.880533589s) [seg_pos={0 0}, upload=447260]: Done begin upload segment. Dec 26 21:10:14.009487995 (509h41m13.88057103s) [noise=true , address=145.79.227.39:4015 , piece_id=7YYK7YEKGG7RWAQCULDDR5MDQEEFHNVQJ2TXAGGBWG64WZDZCNJQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.009544302 (509h41m13.880627348s) [noise=true , address=145.79.227.49:4339 , piece_id=MDEW75R7XIVL7Z72OQENNPMQRT2UF34AD4OGVBKC7SV5G4GNNNAQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.010025519 (509h41m13.881108564s) [noise=true , address=145.79.227.25:4618 , piece_id=7IUHJ2A2EIVYSAHT7FT246PKZ7X6MEHR2KPEEBE7FX3LCAPKT6TA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.010472681 (509h41m13.881555726s) [noise=true , address=77.90.31.21:4878 , piece_id=G5MJD2PRQY7BF24AHVFHCAKT2DFMHF4SDJ4PMKUATHPGVV5W4VPA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.010877461 (509h41m13.881960496s) [noise=true , address=145.79.227.42:4123 , piece_id=AZXKUV6DKAYU6S56LEHT2UNMBDHICNME7EQWOOODU7AT52FIA3HA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.011277152 (509h41m13.882360197s) [noise=true , address=145.79.227.47:4265 , piece_id=KYMX5GGAWG3BSAVYCGV6ROW25A5FRLQOUWL55HGN7OVPURIWC4AA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.011682604 (509h41m13.882765639s) [noise=true , address=145.79.227.51:4409 , piece_id=CJXKGFMM53PDOXUSWP4ET6P5WX3XOR5AHOKYFPMSM52HFAD37LQA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.012140527 (509h41m13.883223572s) [noise=true , address=145.79.227.2:4115 , piece_id=3EVCYQCJS6WDEMMU3DW7BY3P6EJMLSDKS56QFCO4NNF7VD2TH7CA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.012148331 (509h41m13.883231376s) [noise=true , address=45.91.116.6:4973 , piece_id=V7OKX355P5DPK6MLAJBLVVBPXWQKJB3RVF6NPOQARA2XQMETUMRA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.012650307 (509h41m13.883733352s) [noise=true , address=145.79.227.45:4230 , piece_id=DYTIPRJNAIYMEEHVC6II72OX3V2F5XI6EWKYFGK7UZS423ZEH5YA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.013164566 (509h41m13.884247611s) [noise=true , address=77.90.31.14:4662 , piece_id=QYN3DR6AXCX44QSXYZBAFD77E4QLRKM3YELGPJWUUCXRCLVZIW4A, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.013663305 (509h41m13.88474635s) [noise=true , address=145.79.227.19:4079 , piece_id=UH7DXIECNS4MKQ5DHKHC75IEPUGPFXM7IXMQK4XCV5DL4IUO5DKA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.014188425 (509h41m13.88527147s) [noise=true , address=145.79.227.13:4804 , piece_id=MRCJV4NHQ4XL744MF7NJ662ZKOSQYVO6ZINT3ASVBYSFFPTVIXSQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.014397063 (509h41m13.885480098s) [noise=true , address=145.79.227.37:4566 , piece_id=GEE2DOJP2AYMMACC7XTBDYSDKKI5I2NLFNOULZMKEU7FP6XOW6IQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.014914969 (509h41m13.885998014s) [noise=true , address=77.90.31.15:4700 , piece_id=CHTNLRWXIZ345CGMSZLJDTPSLMPX2O4J44RNUBWCM6R672SU55HA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.015436001 (509h41m13.886519036s) [noise=true , address=77.90.31.18:4808 , piece_id=3KRCQKHZZB4N2KXNNWWZT62PL75NNB4WALATANY6KPOUMQS4BQHA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:15.722767878 (509h41m15.593850914s) [upload=447260]: Next returned nil segment Dec 26 21:10:15.722785863 (509h41m15.593868908s) [upload=447260]: Waiting for error group managing segments... Dec 26 21:10:16.006348673 (509h41m15.877431708s) [noise=true , address=145.79.227.47:4265 , piece_id=KYMX5GGAWG3BSAVYCGV6ROW25A5FRLQOUWL55HGN7OVPURIWC4AA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.016267210 (509h41m15.887350245s) [noise=true , address=145.79.227.39:4015 , piece_id=7YYK7YEKGG7RWAQCULDDR5MDQEEFHNVQJ2TXAGGBWG64WZDZCNJQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.021858436 (509h41m15.892941481s) [noise=true , address=145.79.227.42:4123 , piece_id=AZXKUV6DKAYU6S56LEHT2UNMBDHICNME7EQWOOODU7AT52FIA3HA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.025907927 (509h41m15.896990962s) [noise=true , address=145.79.227.45:4230 , piece_id=DYTIPRJNAIYMEEHVC6II72OX3V2F5XI6EWKYFGK7UZS423ZEH5YA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.032766748 (509h41m15.903849793s) [noise=true , address=145.79.227.37:4566 , piece_id=GEE2DOJP2AYMMACC7XTBDYSDKKI5I2NLFNOULZMKEU7FP6XOW6IQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.036155470 (509h41m15.907238515s) [noise=true , address=145.79.227.2:4115 , piece_id=3EVCYQCJS6WDEMMU3DW7BY3P6EJMLSDKS56QFCO4NNF7VD2TH7CA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.038847056 (509h41m15.909930091s) [noise=true , address=145.79.227.51:4409 , piece_id=CJXKGFMM53PDOXUSWP4ET6P5WX3XOR5AHOKYFPMSM52HFAD37LQA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.054354786 (509h41m15.925437821s) [noise=true , address=145.79.227.49:4339 , piece_id=MDEW75R7XIVL7Z72OQENNPMQRT2UF34AD4OGVBKC7SV5G4GNNNAQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.119769168 (509h41m15.990852213s) [noise=true , address=145.79.227.25:4618 , piece_id=7IUHJ2A2EIVYSAHT7FT246PKZ7X6MEHR2KPEEBE7FX3LCAPKT6TA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.186031665 (509h41m16.0571147s) [noise=true , address=77.90.31.15:4700 , piece_id=CHTNLRWXIZ345CGMSZLJDTPSLMPX2O4J44RNUBWCM6R672SU55HA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.321054485 (509h41m16.19213752s) [noise=true , address=77.90.31.21:4878 , piece_id=G5MJD2PRQY7BF24AHVFHCAKT2DFMHF4SDJ4PMKUATHPGVV5W4VPA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.343640422 (509h41m16.214723467s) [noise=true , address=45.91.116.6:4973 , piece_id=V7OKX355P5DPK6MLAJBLVVBPXWQKJB3RVF6NPOQARA2XQMETUMRA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:17.452739637 (509h41m17.323822672s) [noise=true , address=145.79.227.13:4804 , piece_id=MRCJV4NHQ4XL744MF7NJ662ZKOSQYVO6ZINT3ASVBYSFFPTVIXSQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:17.452768592 (509h41m17.323851637s) [seg_pos={0 0}, upload=447260]: Segment reached optimal threshold of 13 pieces. Dec 26 21:10:17.453036703 (509h41m17.324119768s) [noise=true , address=77.90.31.18:4808 , piece_id=3KRCQKHZZB4N2KXNNWWZT62PL75NNB4WALATANY6KPOUMQS4BQHA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 21:10:17.453047553 (509h41m17.324130598s) [noise=true , address=77.90.31.18:4808 , piece_id=3KRCQKHZZB4N2KXNNWWZT62PL75NNB4WALATANY6KPOUMQS4BQHA, seg_pos={0 0}, upload=447260]: Long tail cancellation due to optimal threshold Dec 26 21:10:17.453097659 (509h41m17.324180694s) [noise=true , address=145.79.227.19:4079 , piece_id=UH7DXIECNS4MKQ5DHKHC75IEPUGPFXM7IXMQK4XCV5DL4IUO5DKA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:53268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:53268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53268->145.79.227.19:4079: use of closed network connection Dec 26 21:10:17.453105835 (509h41m17.32418887s) [noise=true , address=145.79.227.19:4079 , piece_id=UH7DXIECNS4MKQ5DHKHC75IEPUGPFXM7IXMQK4XCV5DL4IUO5DKA, seg_pos={0 0}, upload=447260]: Long tail cancellation due to optimal threshold Dec 26 21:10:17.453344669 (509h41m17.324427725s) [noise=true , address=77.90.31.14:4662 , piece_id=QYN3DR6AXCX44QSXYZBAFD77E4QLRKM3YELGPJWUUCXRCLVZIW4A, seg_pos={0 0}, upload=447260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 21:10:17.453354508 (509h41m17.324437543s) [noise=true , address=77.90.31.14:4662 , piece_id=QYN3DR6AXCX44QSXYZBAFD77E4QLRKM3YELGPJWUUCXRCLVZIW4A, seg_pos={0 0}, upload=447260]: Long tail cancellation due to optimal threshold Dec 26 21:10:17.453389505 (509h41m17.32447254s) [seg_pos={0 0}, upload=447260]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:10:17.453486840 (509h41m17.324569885s) [upload=447260]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:10:17.453492801 (509h41m17.324575836s) [upload=447260]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:10:17.766755955 (509h41m17.63783901s) [upload=447260]: Done uploading segments... Dec 26 21:10:17.766779971 (509h41m17.637863016s) [upload=447260]: Upload finished. err: <nil> Duration: 27.038214613s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....................................X.....................] duration: 32.979023678s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:11:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 32.979042264s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...XX.XXX...X.XX..XXX...............X....................XX] duration: 30.003787872s trace: https://jaeger.dev.storj.io/trace/412ab68d06408eac FAIL (2025-12-26 21:10:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 21:10:13.756848981 (509h41m13.627932016s) [upload=447260]: Starting upload Dec 26 21:10:13.756907482 (509h41m13.627990517s) [upload=447260]: Done starting upload Dec 26 21:10:13.757014996 (509h41m13.628098031s) [upload=447260]: Uploading segments... Dec 26 21:10:13.757094799 (509h41m13.628177834s) [upload=447260]: Got next segment. Inline: false Dec 26 21:10:13.757133873 (509h41m13.628216918s) [upload=447260]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:10:13.757140265 (509h41m13.62822331s) [upload=447260]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:10:14.009399737 (509h41m13.880482792s) [seg_pos={0 0}, upload=447260]: Begin upload segment... Dec 26 21:10:14.009450544 (509h41m13.880533589s) [seg_pos={0 0}, upload=447260]: Done begin upload segment. Dec 26 21:10:14.009487995 (509h41m13.88057103s) [noise=true , address=145.79.227.39:4015 , piece_id=7YYK7YEKGG7RWAQCULDDR5MDQEEFHNVQJ2TXAGGBWG64WZDZCNJQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.009544302 (509h41m13.880627348s) [noise=true , address=145.79.227.49:4339 , piece_id=MDEW75R7XIVL7Z72OQENNPMQRT2UF34AD4OGVBKC7SV5G4GNNNAQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.010025519 (509h41m13.881108564s) [noise=true , address=145.79.227.25:4618 , piece_id=7IUHJ2A2EIVYSAHT7FT246PKZ7X6MEHR2KPEEBE7FX3LCAPKT6TA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.010472681 (509h41m13.881555726s) [noise=true , address=77.90.31.21:4878 , piece_id=G5MJD2PRQY7BF24AHVFHCAKT2DFMHF4SDJ4PMKUATHPGVV5W4VPA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.010877461 (509h41m13.881960496s) [noise=true , address=145.79.227.42:4123 , piece_id=AZXKUV6DKAYU6S56LEHT2UNMBDHICNME7EQWOOODU7AT52FIA3HA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.011277152 (509h41m13.882360197s) [noise=true , address=145.79.227.47:4265 , piece_id=KYMX5GGAWG3BSAVYCGV6ROW25A5FRLQOUWL55HGN7OVPURIWC4AA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.011682604 (509h41m13.882765639s) [noise=true , address=145.79.227.51:4409 , piece_id=CJXKGFMM53PDOXUSWP4ET6P5WX3XOR5AHOKYFPMSM52HFAD37LQA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.012140527 (509h41m13.883223572s) [noise=true , address=145.79.227.2:4115 , piece_id=3EVCYQCJS6WDEMMU3DW7BY3P6EJMLSDKS56QFCO4NNF7VD2TH7CA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.012148331 (509h41m13.883231376s) [noise=true , address=45.91.116.6:4973 , piece_id=V7OKX355P5DPK6MLAJBLVVBPXWQKJB3RVF6NPOQARA2XQMETUMRA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.012650307 (509h41m13.883733352s) [noise=true , address=145.79.227.45:4230 , piece_id=DYTIPRJNAIYMEEHVC6II72OX3V2F5XI6EWKYFGK7UZS423ZEH5YA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.013164566 (509h41m13.884247611s) [noise=true , address=77.90.31.14:4662 , piece_id=QYN3DR6AXCX44QSXYZBAFD77E4QLRKM3YELGPJWUUCXRCLVZIW4A, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.013663305 (509h41m13.88474635s) [noise=true , address=145.79.227.19:4079 , piece_id=UH7DXIECNS4MKQ5DHKHC75IEPUGPFXM7IXMQK4XCV5DL4IUO5DKA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.014188425 (509h41m13.88527147s) [noise=true , address=145.79.227.13:4804 , piece_id=MRCJV4NHQ4XL744MF7NJ662ZKOSQYVO6ZINT3ASVBYSFFPTVIXSQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.014397063 (509h41m13.885480098s) [noise=true , address=145.79.227.37:4566 , piece_id=GEE2DOJP2AYMMACC7XTBDYSDKKI5I2NLFNOULZMKEU7FP6XOW6IQ, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.014914969 (509h41m13.885998014s) [noise=true , address=77.90.31.15:4700 , piece_id=CHTNLRWXIZ345CGMSZLJDTPSLMPX2O4J44RNUBWCM6R672SU55HA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:14.015436001 (509h41m13.886519036s) [noise=true , address=77.90.31.18:4808 , piece_id=3KRCQKHZZB4N2KXNNWWZT62PL75NNB4WALATANY6KPOUMQS4BQHA, seg_pos={0 0}, upload=447260]: Uploading piece... Dec 26 21:10:15.722767878 (509h41m15.593850914s) [upload=447260]: Next returned nil segment Dec 26 21:10:15.722785863 (509h41m15.593868908s) [upload=447260]: Waiting for error group managing segments... Dec 26 21:10:16.006348673 (509h41m15.877431708s) [noise=true , address=145.79.227.47:4265 , piece_id=KYMX5GGAWG3BSAVYCGV6ROW25A5FRLQOUWL55HGN7OVPURIWC4AA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.016267210 (509h41m15.887350245s) [noise=true , address=145.79.227.39:4015 , piece_id=7YYK7YEKGG7RWAQCULDDR5MDQEEFHNVQJ2TXAGGBWG64WZDZCNJQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.021858436 (509h41m15.892941481s) [noise=true , address=145.79.227.42:4123 , piece_id=AZXKUV6DKAYU6S56LEHT2UNMBDHICNME7EQWOOODU7AT52FIA3HA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.025907927 (509h41m15.896990962s) [noise=true , address=145.79.227.45:4230 , piece_id=DYTIPRJNAIYMEEHVC6II72OX3V2F5XI6EWKYFGK7UZS423ZEH5YA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.032766748 (509h41m15.903849793s) [noise=true , address=145.79.227.37:4566 , piece_id=GEE2DOJP2AYMMACC7XTBDYSDKKI5I2NLFNOULZMKEU7FP6XOW6IQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.036155470 (509h41m15.907238515s) [noise=true , address=145.79.227.2:4115 , piece_id=3EVCYQCJS6WDEMMU3DW7BY3P6EJMLSDKS56QFCO4NNF7VD2TH7CA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.038847056 (509h41m15.909930091s) [noise=true , address=145.79.227.51:4409 , piece_id=CJXKGFMM53PDOXUSWP4ET6P5WX3XOR5AHOKYFPMSM52HFAD37LQA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.054354786 (509h41m15.925437821s) [noise=true , address=145.79.227.49:4339 , piece_id=MDEW75R7XIVL7Z72OQENNPMQRT2UF34AD4OGVBKC7SV5G4GNNNAQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.119769168 (509h41m15.990852213s) [noise=true , address=145.79.227.25:4618 , piece_id=7IUHJ2A2EIVYSAHT7FT246PKZ7X6MEHR2KPEEBE7FX3LCAPKT6TA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.186031665 (509h41m16.0571147s) [noise=true , address=77.90.31.15:4700 , piece_id=CHTNLRWXIZ345CGMSZLJDTPSLMPX2O4J44RNUBWCM6R672SU55HA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.321054485 (509h41m16.19213752s) [noise=true , address=77.90.31.21:4878 , piece_id=G5MJD2PRQY7BF24AHVFHCAKT2DFMHF4SDJ4PMKUATHPGVV5W4VPA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:16.343640422 (509h41m16.214723467s) [noise=true , address=45.91.116.6:4973 , piece_id=V7OKX355P5DPK6MLAJBLVVBPXWQKJB3RVF6NPOQARA2XQMETUMRA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:17.452739637 (509h41m17.323822672s) [noise=true , address=145.79.227.13:4804 , piece_id=MRCJV4NHQ4XL744MF7NJ662ZKOSQYVO6ZINT3ASVBYSFFPTVIXSQ, seg_pos={0 0}, upload=447260]: Done uploading piece. err: <nil> Dec 26 21:10:17.452768592 (509h41m17.323851637s) [seg_pos={0 0}, upload=447260]: Segment reached optimal threshold of 13 pieces. Dec 26 21:10:17.453036703 (509h41m17.324119768s) [noise=true , address=77.90.31.18:4808 , piece_id=3KRCQKHZZB4N2KXNNWWZT62PL75NNB4WALATANY6KPOUMQS4BQHA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 26 21:10:17.453047553 (509h41m17.324130598s) [noise=true , address=77.90.31.18:4808 , piece_id=3KRCQKHZZB4N2KXNNWWZT62PL75NNB4WALATANY6KPOUMQS4BQHA, seg_pos={0 0}, upload=447260]: Long tail cancellation due to optimal threshold Dec 26 21:10:17.453097659 (509h41m17.324180694s) [noise=true , address=145.79.227.19:4079 , piece_id=UH7DXIECNS4MKQ5DHKHC75IEPUGPFXM7IXMQK4XCV5DL4IUO5DKA, seg_pos={0 0}, upload=447260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:53268->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:53268->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53268->145.79.227.19:4079: use of closed network connection Dec 26 21:10:17.453105835 (509h41m17.32418887s) [noise=true , address=145.79.227.19:4079 , piece_id=UH7DXIECNS4MKQ5DHKHC75IEPUGPFXM7IXMQK4XCV5DL4IUO5DKA, seg_pos={0 0}, upload=447260]: Long tail cancellation due to optimal threshold Dec 26 21:10:17.453344669 (509h41m17.324427725s) [noise=true , address=77.90.31.14:4662 , piece_id=QYN3DR6AXCX44QSXYZBAFD77E4QLRKM3YELGPJWUUCXRCLVZIW4A, seg_pos={0 0}, upload=447260]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 26 21:10:17.453354508 (509h41m17.324437543s) [noise=true , address=77.90.31.14:4662 , piece_id=QYN3DR6AXCX44QSXYZBAFD77E4QLRKM3YELGPJWUUCXRCLVZIW4A, seg_pos={0 0}, upload=447260]: Long tail cancellation due to optimal threshold Dec 26 21:10:17.453389505 (509h41m17.32447254s) [seg_pos={0 0}, upload=447260]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:10:17.453486840 (509h41m17.324569885s) [upload=447260]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:10:17.453492801 (509h41m17.324575836s) [upload=447260]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:10:17.766755955 (509h41m17.63783901s) [upload=447260]: Done uploading segments... Dec 26 21:10:17.766779971 (509h41m17.637863016s) [upload=447260]: Upload finished. err: <nil> Duration: 30.003812117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.....................................X....................] duration: 34.963591249s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:10:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 34.963648118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X...XX.XXX...X.XX..XXX...............X....................X] duration: 30.002803262s trace: https://jaeger.dev.storj.io/trace/2c3d8ae2464df012 FAIL (2025-12-26 21:10:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: 24X2R55HG2NONFDJMMAT5FAGU77W5CHPTFUEQOYA4VQA7RTU5QVA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 3NU5ZMGN6JHHPFVXR7XDMKDMCKC3TUHBT5OBSDM34IEU2G7OPQHA) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: 6HSTDKRVVQRP6UHAZQA4GMENSFZ4JFDHCN54M6OBZRVOMS726ZGA) context deadline exceeded; piecestore: (Node ID: 12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, Piece ID: XKT44DDNYZ3P752QFK7EX2Q5LXN3WD4JKDVZKW3WZQWXDQ2ZO3CQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: VLDFWHBCCR3H6JD3GOUZ5NPCXF4WVPZ7UJJZEVLOZMDN4WG5DADA) context deadline exceeded ======LOGS====== .66404973s) [upload=447252]: Starting upload Dec 26 21:09:42.793052078 (509h40m42.664135113s) [upload=447252]: Done starting upload Dec 26 21:09:42.793138523 (509h40m42.664221558s) [upload=447252]: Uploading segments... Dec 26 21:09:42.793272568 (509h40m42.664355603s) [upload=447252]: Got next segment. Inline: false Dec 26 21:09:42.793304078 (509h40m42.664387113s) [upload=447252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 21:09:42.793308757 (509h40m42.664391792s) [upload=447252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 21:09:43.039972091 (509h40m42.911055126s) [seg_pos={0 0}, upload=447252]: Begin upload segment... Dec 26 21:09:43.040025343 (509h40m42.911108378s) [seg_pos={0 0}, upload=447252]: Done begin upload segment. Dec 26 21:09:43.040082131 (509h40m42.911165166s) [noise=true , address=45.91.116.5:4300 , piece_id=3NU5ZMGN6JHHPFVXR7XDMKDMCKC3TUHBT5OBSDM34IEU2G7OPQHA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.040091408 (509h40m42.911174464s) [noise=true , address=145.79.227.4:4185 , piece_id=MZ7CU732P5EANRCZ62VO4EWYRAGLDFVF7RRIGHMX6J6YKWAAU63Q, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.040537498 (509h40m42.911620533s) [noise=true , address=145.79.227.42:4121 , piece_id=VLDFWHBCCR3H6JD3GOUZ5NPCXF4WVPZ7UJJZEVLOZMDN4WG5DADA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.040544551 (509h40m42.911627596s) [noise=true , address=145.79.227.13:4803 , piece_id=XKT44DDNYZ3P752QFK7EX2Q5LXN3WD4JKDVZKW3WZQWXDQ2ZO3CQ, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.040942018 (509h40m42.912025053s) [noise=true , address=145.79.227.45:4231 , piece_id=HAJZIDNNJRAG6FRQQCNZDHBAIRH2R3ZFATTO5UBDK2DU6HQQJ7LA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.041214597 (509h40m42.912297632s) [noise=true , address=45.91.116.10:4082 , piece_id=23S7SYII3YSGIRD7764I5BQDA6MAZYF4AV7W3KQUECCKNZALK6DA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.041478780 (509h40m42.912561815s) [noise=true , address=145.79.227.39:4014 , piece_id=TNXIVGYKCGHWQ56VCKDGEPHDUMGGPZX7LGORDYFWG7CVPCNQHZFA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.041772179 (509h40m42.912855214s) [noise=true , address=77.90.31.15:4700 , piece_id=RCXLATDNTMVIJJOGFAEYFFIXGYNXEHMOKTQQAJ5QOBBZMPZVITDQ, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.042072260 (509h40m42.913155295s) [noise=true , address=145.79.227.15:4876 , piece_id=P5Z7ARIN5LRRY2IK4XZRJXALW3ZP2AUM5KDJOZDJ5LLIV5TWTRSA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.042615425 (509h40m42.91369846s) [noise=true , address=45.91.116.12:4227 , piece_id=RIVTY575UQHGNPYLXE3ADGIS3JNW6AELKPJNMGASUS7I4WGVRUBA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.043006479 (509h40m42.914089525s) [noise=true , address=45.91.116.6:4974 , piece_id=24X2R55HG2NONFDJMMAT5FAGU77W5CHPTFUEQOYA4VQA7RTU5QVA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.043365573 (509h40m42.914448608s) [noise=true , address=45.91.116.7:4009 , piece_id=54PRVZVZP65FNGU4LWPY3B4BS6PMUN24ZX2EEPCYQ37EWR4Y3BXQ, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.044465708 (509h40m42.915548753s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQXA4DJH32M5NULK3SP43CDDBNDFX7AMIRB65R674C3VKO6QMLFA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.044864778 (509h40m42.915947813s) [noise=true , address=77.90.31.21:4880 , piece_id=6HSTDKRVVQRP6UHAZQA4GMENSFZ4JFDHCN54M6OBZRVOMS726ZGA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.046432494 (509h40m42.917515539s) [noise=true , address=45.91.116.13:4153 , piece_id=LCOIFI5KRFEOSK47RWGYJYWMNXEX36RBP7B6EIPA3DBUIZCZEMWQ, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:43.048932233 (509h40m42.920015278s) [noise=true , address=145.79.227.52:4447 , piece_id=CHC3JQGGC7LEDYNKV3GMWAAYM6MTP6S7743UQ662C3T255UHK4QA, seg_pos={0 0}, upload=447252]: Uploading piece... Dec 26 21:09:44.756722393 (509h40m44.627805428s) [upload=447252]: Next returned nil segment Dec 26 21:09:44.756742612 (509h40m44.627825657s) [upload=447252]: Waiting for error group managing segments... Dec 26 21:09:45.046203241 (509h40m44.917286287s) [noise=true , address=145.79.227.52:4447 , piece_id=CHC3JQGGC7LEDYNKV3GMWAAYM6MTP6S7743UQ662C3T255UHK4QA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.046484798 (509h40m44.917567843s) [noise=true , address=145.79.227.15:4876 , piece_id=P5Z7ARIN5LRRY2IK4XZRJXALW3ZP2AUM5KDJOZDJ5LLIV5TWTRSA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.055068012 (509h40m44.926151047s) [noise=true , address=145.79.227.42:4121 , piece_id=VLDFWHBCCR3H6JD3GOUZ5NPCXF4WVPZ7UJJZEVLOZMDN4WG5DADA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.068091280 (509h40m44.939174325s) [noise=true , address=145.79.227.39:4014 , piece_id=TNXIVGYKCGHWQ56VCKDGEPHDUMGGPZX7LGORDYFWG7CVPCNQHZFA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.074768335 (509h40m44.94585137s) [noise=true , address=145.79.227.46:4602 , piece_id=ZQXA4DJH32M5NULK3SP43CDDBNDFX7AMIRB65R674C3VKO6QMLFA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.078893821 (509h40m44.949976856s) [noise=true , address=145.79.227.45:4231 , piece_id=HAJZIDNNJRAG6FRQQCNZDHBAIRH2R3ZFATTO5UBDK2DU6HQQJ7LA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.137189437 (509h40m45.008272472s) [noise=true , address=145.79.227.13:4803 , piece_id=XKT44DDNYZ3P752QFK7EX2Q5LXN3WD4JKDVZKW3WZQWXDQ2ZO3CQ, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.296531415 (509h40m45.16761445s) [noise=true , address=45.91.116.6:4974 , piece_id=24X2R55HG2NONFDJMMAT5FAGU77W5CHPTFUEQOYA4VQA7RTU5QVA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.333748364 (509h40m45.204831409s) [noise=true , address=77.90.31.21:4880 , piece_id=6HSTDKRVVQRP6UHAZQA4GMENSFZ4JFDHCN54M6OBZRVOMS726ZGA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.367409121 (509h40m45.238492166s) [noise=true , address=45.91.116.7:4009 , piece_id=54PRVZVZP65FNGU4LWPY3B4BS6PMUN24ZX2EEPCYQ37EWR4Y3BXQ, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.410872113 (509h40m45.281955148s) [noise=true , address=45.91.116.5:4300 , piece_id=3NU5ZMGN6JHHPFVXR7XDMKDMCKC3TUHBT5OBSDM34IEU2G7OPQHA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.481362762 (509h40m45.352445797s) [noise=true , address=77.90.31.15:4700 , piece_id=RCXLATDNTMVIJJOGFAEYFFIXGYNXEHMOKTQQAJ5QOBBZMPZVITDQ, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.483317604 (509h40m45.354400649s) [noise=true , address=45.91.116.10:4082 , piece_id=23S7SYII3YSGIRD7764I5BQDA6MAZYF4AV7W3KQUECCKNZALK6DA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: <nil> Dec 26 21:09:45.483329867 (509h40m45.354412912s) [seg_pos={0 0}, upload=447252]: Segment reached optimal threshold of 13 pieces. Dec 26 21:09:45.483543183 (509h40m45.354626248s) [noise=true , address=45.91.116.12:4227 , piece_id=RIVTY575UQHGNPYLXE3ADGIS3JNW6AELKPJNMGASUS7I4WGVRUBA, seg_pos={0 0}, upload=447252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 26 21:09:45.483551620 (509h40m45.354634665s) [noise=true , address=45.91.116.12:4227 , piece_id=RIVTY575UQHGNPYLXE3ADGIS3JNW6AELKPJNMGASUS7I4WGVRUBA, seg_pos={0 0}, upload=447252]: Long tail cancellation due to optimal threshold Dec 26 21:09:45.483799932 (509h40m45.354882987s) [noise=true , address=45.91.116.13:4153 , piece_id=LCOIFI5KRFEOSK47RWGYJYWMNXEX36RBP7B6EIPA3DBUIZCZEMWQ, seg_pos={0 0}, upload=447252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: rpc: dial tcp 45.91.116.13:4153: operation was canceled; conn closed: op Dec 26 21:09:45.483807607 (509h40m45.354890652s) [noise=true , address=45.91.116.13:4153 , piece_id=LCOIFI5KRFEOSK47RWGYJYWMNXEX36RBP7B6EIPA3DBUIZCZEMWQ, seg_pos={0 0}, upload=447252]: Long tail cancellation due to optimal threshold Dec 26 21:09:45.484003360 (509h40m45.355086415s) [noise=true , address=145.79.227.4:4185 , piece_id=MZ7CU732P5EANRCZ62VO4EWYRAGLDFVF7RRIGHMX6J6YKWAAU63Q, seg_pos={0 0}, upload=447252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:48058->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:48058->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48058->145.79.227.4:4185: use of closed network connection Dec 26 21:09:45.484020742 (509h40m45.355103778s) [noise=true , address=145.79.227.4:4185 , piece_id=MZ7CU732P5EANRCZ62VO4EWYRAGLDFVF7RRIGHMX6J6YKWAAU63Q, seg_pos={0 0}, upload=447252]: Long tail cancellation due to optimal threshold Dec 26 21:09:45.484044057 (509h40m45.355127092s) [seg_pos={0 0}, upload=447252]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 21:09:45.484129439 (509h40m45.355212484s) [upload=447252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 21:09:45.484135962 (509h40m45.355218997s) [upload=447252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 21:09:45.783522824 (509h40m45.654605859s) [upload=447252]: Done uploading segments... Dec 26 21:09:45.783540127 (509h40m45.654623172s) [upload=447252]: Upload finished. err: <nil> Duration: 30.00282822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.....................................X...................] duration: 31.581313721s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:09:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 31.581375548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.....................................X..................] duration: 31.791548215s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 31.791579194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X...XX.XXX...X.XX..XXX...............X....................] duration: 11.595649722s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:08:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 11.595674017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X...XX.XXX...X.XX..XXX...............X...................] duration: 27.216686268s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:08:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 27.216715122s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.....................................X.................] duration: 11.184393096s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:08:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 11.184423985s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.X...XX.XXX...X.XX..XXX...............X..................] duration: 22.636773268s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:07:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 22.636798074s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.....................................X................] duration: 15.468587766s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:07:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.468603006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.....................................X...............] duration: 47.758725744s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:06:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 47.758753898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.X...XX.XXX...X.XX..XXX...............X.................] duration: 7.519094723s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:06:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 7.519120593s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...X.X...XX.XXX...X.XX..XXX...............X................] duration: 6.93553583s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:05:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.935562419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.....................................X..............] duration: 15.19084519s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:05:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.190870859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.....................................X.............] duration: 17.397435285s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:05:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 17.397467986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X...X.X...XX.XXX...X.XX..XXX...............X...............] duration: 8.781062266s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 8.781087455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.....................................X............] duration: 23.883766061s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:04:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 23.883808511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X...X.X...XX.XXX...X.XX..XXX...............X..............] duration: 9.44396444s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:03:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 9.444004867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.....................................X...........] duration: 7.55850535s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:03:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 7.558531909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X...X.X...XX.XXX...X.XX..XXX...............X.............] duration: 6.372298924s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:03:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.372323962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XXX.X...X.X...XX.XXX...X.XX..XXX...............X............] duration: 7.829574977s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:02:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 7.829603732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.....................................X..........] duration: 8.159669972s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:02:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 8.159701261s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XXXX.X...X.X...XX.XXX...X.XX..XXX...............X...........] duration: 6.151109038s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:02:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.151161198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXXX.X...X.X...XX.XXX...X.XX..XXX...............X..........] duration: 6.030873028s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:01:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.030927631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.....................................X.........] duration: 6.8359562s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:01:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 6.835989493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X.........] duration: 6.755668947s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.755694737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X........] duration: 6.05307507s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.053106589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.....................................X........] duration: 15.408945478s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 21:00:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.408973361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X.......] duration: 28.633710419s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 21:00:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 28.633761407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.....................................X.......] duration: 20.622887639s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:59:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 20.622905333s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X......] duration: 8.542450222s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:59:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 8.542464481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X.....] duration: 22.928546266s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:59:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 22.928580882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..............X.....................................X......] duration: 25.982273055s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:59:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 25.98230817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..............X.....................................X.....] duration: 30.241158009s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:58:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 30.241186995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X....] duration: 20.576520069s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:58:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 20.576548022s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..............X.....................................X....] duration: 16.267512243s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:57:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 16.267544034s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X...] duration: 6.168482396s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:57:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 6.168514438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X..] duration: 10.89577914s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:56:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 10.895807935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..............X.....................................X...] duration: 9.466243249s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 9.466267725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..............X.....................................X..] duration: 15.294929525s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:55:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 15.294955754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X.] duration: 22.576799698s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:55:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 22.576820648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..............X.....................................X.] duration: 6.66081701s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:55:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 6.660833421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..............X.....................................X] duration: 1m0.003667736s trace: https://jaeger.dev.storj.io/trace/6d2950019117adaf FAIL (2025-12-26 20:55:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1Rj4hUGWvUPuU49QDsHYhibiJhyY5KDGus1rzNeP7KusgyvTsU, Piece ID: NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A) context deadline exceeded; piecestore: (Node ID: 12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, Piece ID: N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ) context deadline exceeded; piecestore: (Node ID: 12N61Va6p4WU9cmUz65CkCDKr54LFtJhh8HJGYPVt8cJTRoWLhb, Piece ID: IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ) context deadline exceeded ======LOGS====== ng piece... Dec 26 20:54:08.087336805 (509h25m7.95841984s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087649520 (509h25m7.958732565s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.087988717 (509h25m7.959071752s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088323233 (509h25m7.959406268s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.088877028 (509h25m7.959960063s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089247203 (509h25m7.960330238s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089543066 (509h25m7.960626101s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.089914133 (509h25m7.960997168s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090327259 (509h25m7.961410294s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.090645345 (509h25m7.96172838s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091004318 (509h25m7.962087353s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091338976 (509h25m7.962422021s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:08.091732054 (509h25m7.962815099s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Uploading piece... Dec 26 20:54:09.565187143 (509h25m9.436270188s) [upload=447022]: Next returned nil segment Dec 26 20:54:09.565201321 (509h25m9.436284356s) [upload=447022]: Waiting for error group managing segments... Dec 26 20:54:10.137065972 (509h25m10.008149007s) [noise=true , address=77.90.31.10:4523 , piece_id=GQYT3Z53Y6HVTY3IBUNFCIWBEE7KVAUWEJYO7QJFRRAJCHTJMYFA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.137147197 (509h25m10.008230232s) [noise=true , address=77.90.31.12:4618 , piece_id=2FN7JRVK3HK5HHNCPKMVUJRQNJJACMGE24D4ZP7AFPOOJBKMATTQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.149325196 (509h25m10.020408242s) [noise=true , address=145.79.227.51:4438 , piece_id=DFMUHLKLMUHZT3DB332HUSYPWSMHI5BMLLSTR5ZDMJYCRGGIVVWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.159611673 (509h25m10.030694718s) [noise=true , address=145.79.227.26:4681 , piece_id=A5PV6SGPJAJG3JPDKJHBV2VTME4EIACM3IIMJBIC2IIRES4WNIPQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.164411764 (509h25m10.035494809s) [noise=true , address=145.79.227.23:4568 , piece_id=NSLFF5J4ALCMVJH52RQP2ZYJWC233W4X3JFX552MKWCFBJ6PM45A, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.190924278 (509h25m10.062007313s) [noise=true , address=45.91.116.12:4242 , piece_id=LQAQRHVMRU2M3JV2LYWDID3DOXUWOXYWIW7ZEKQPQXLJG7IOEAEQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.232931086 (509h25m10.104014131s) [noise=true , address=77.90.31.14:4680 , piece_id=GIO7UID4U4FT5EM7PNJLDF3UFVZXXMHCQQZ6WLDDQGND4F25ZFRQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.241658245 (509h25m10.11274129s) [noise=true , address=77.90.31.13:4643 , piece_id=7JR4DX45KRLEWATSJY7MKS6LGPYESBXPK4XXYMRD6Y5PGE2MY5KA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.472574370 (509h25m10.343657405s) [noise=true , address=145.79.227.47:4293 , piece_id=5KSF3ACAOLWMD53UAA6KVZ3KYER5LXJ3VGBYHGRS6SONHUVDNK4Q, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.866910566 (509h25m10.737993611s) [noise=true , address=145.79.227.45:4251 , piece_id=3XUB6BZUXFZFCGRYCBBAXUHT42UB5RTQLGZ7UZ5RVAZGIUAVCARA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:10.886583507 (509h25m10.757666542s) [noise=true , address=145.79.227.15:4884 , piece_id=6XNGEMFFW4IAVKKQBCCQQ6J4CMGTOMCGU4OE2P5VCW4BG5DBMQMA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.127773215 (509h25m10.99885626s) [noise=true , address=145.79.227.44:4225 , piece_id=N5STVBKS2B3Y6WNEARP6FB3VFL7YVSK6H33E5ANR2EJFZPXMMLJQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.192133217 (509h25m11.063216263s) [noise=true , address=145.79.227.13:4819 , piece_id=BQDRL2QJZ75TAMUM2VSGTF4U3NEMZC4FFQLN6QV6UOAFHWB56MZA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256146219 (509h25m11.127229264s) [noise=true , address=145.79.227.40:4062 , piece_id=IDKKYLNPIYXCL2FM7HZCYJOFUP3FZHEHZM6INFLRRSNW6T6BTRSQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: <nil> Dec 26 20:54:11.256176597 (509h25m11.127259632s) [seg_pos={0 0}, upload=447022]: Segment reached optimal threshold of 14 pieces. Dec 26 20:54:11.256428137 (509h25m11.127511192s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dwmwypGjZv8TinKXYSfCEU4z7eSRML4GsXCBypzSRq6WnBs98): protocol: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45378->145.79.227.27:4718: use of closed network connection Dec 26 20:54:11.256439549 (509h25m11.127522584s) [noise=true , address=145.79.227.27:4718 , piece_id=2Q5H4QIXJOKARVVP7ZUDLR3DNZSU4VXDCVPKKBPB5VRWREIYYQXA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256458915 (509h25m11.12754198s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g6e9GSH9aUj8t9T79uLE6p5jPZSnX3cNDLEBHVL88hLqLq2qF): protocol: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36012->145.79.227.27:4713: use of closed network connection Dec 26 20:54:11.256490705 (509h25m11.127573741s) [noise=true , address=145.79.227.27:4713 , piece_id=47PQK3AIASV6ERWWYZ7UPMME4RF3VV6OJDDVD6IAKNW44YHSM5QA, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256698171 (509h25m11.127781216s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: rpc: dial tcp 145.79.227.30:4796: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 20:54:11.256706457 (509h25m11.127789502s) [noise=true , address=145.79.227.30:4796 , piece_id=Q2UKZCNACRW5WX2EG3CU4YT56BUD3LJA6UGQMS6HCMWFCCUZAYWQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256917309 (509h25m11.128000364s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb): protocol: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33198->145.79.227.16:4920: use of closed network connection Dec 26 20:54:11.256926666 (509h25m11.128009711s) [noise=true , address=145.79.227.16:4920 , piece_id=YYDFIS7PPO2BP57SNH7Q6Z47GOEUGG4MUTAYUQRKXCHATDFZI7LQ, seg_pos={0 0}, upload=447022]: Long tail cancellation due to optimal threshold Dec 26 20:54:11.256964869 (509h25m11.128047914s) [seg_pos={0 0}, upload=447022]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 20:54:11.257064138 (509h25m11.128147173s) [upload=447022]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:11.257070690 (509h25m11.128153725s) [upload=447022]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:11.561962810 (509h25m11.433045875s) [upload=447022]: Done uploading segments... Dec 26 20:54:11.561987427 (509h25m11.433070472s) [upload=447022]: Upload finished. err: <nil> Duration: 1m0.00369067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............X] duration: 30.008237729s trace: https://jaeger.dev.storj.io/trace/5bff7ff428a15351 FAIL (2025-12-26 20:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 26 20:54:34.547591131 (509h25m34.418674166s) [upload=447028]: Got next segment. Inline: false Dec 26 20:54:34.547620016 (509h25m34.418703052s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:54:34.547624074 (509h25m34.418707119s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:54:34.795540571 (509h25m34.666623606s) [seg_pos={0 0}, upload=447028]: Begin upload segment... Dec 26 20:54:34.795593181 (509h25m34.666676216s) [seg_pos={0 0}, upload=447028]: Done begin upload segment. Dec 26 20:54:34.795629269 (509h25m34.666712304s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.795656782 (509h25m34.666739817s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796182192 (509h25m34.667265227s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796281672 (509h25m34.667364717s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796551827 (509h25m34.667634862s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796657688 (509h25m34.667740723s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796898527 (509h25m34.667981562s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.796959522 (509h25m34.668042557s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797206573 (509h25m34.668289608s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797448644 (509h25m34.668531679s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.797745459 (509h25m34.668828494s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798103070 (509h25m34.669186105s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798483695 (509h25m34.66956673s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.798833471 (509h25m34.669916506s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799202464 (509h25m34.670285499s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:34.799600581 (509h25m34.670683616s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Uploading piece... Dec 26 20:54:36.671950309 (509h25m36.543033354s) [upload=447028]: Next returned nil segment Dec 26 20:54:36.671966910 (509h25m36.543049955s) [upload=447028]: Waiting for error group managing segments... Dec 26 20:54:36.995584505 (509h25m36.86666754s) [noise=true , address=77.90.31.14:4664 , piece_id=4SUK7GZMIH4K3J7324KORF5WNVPXVLLLXYJD52UUSYBCXKZCJ43A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:36.996225606 (509h25m36.867308641s) [noise=true , address=145.79.227.21:4473 , piece_id=N4PXRUMCYAGXQCP22CRWPEDHNQ6FENDCF746ABDPMLQ4P44WA44A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.003358658 (509h25m36.874441713s) [noise=true , address=145.79.227.46:4601 , piece_id=TQICZMRDIQREMM5WV2TAZM27TVUILA3NLD4ND5NKCHAJIMTKPLJQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.013630438 (509h25m36.884713474s) [noise=true , address=45.91.116.5:4298 , piece_id=67LHNAKM2XKRRDD6MOP5CYVYTCLEJOD3VC6UNF43E55TSNEH3VRA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.035219708 (509h25m36.906302753s) [noise=true , address=45.91.116.6:4973 , piece_id=N5Q7OFZFD475HEU5OM7R2XAMSQCCGFZVOQNAZIGR3FTPOFHQVVBA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.038914914 (509h25m36.909997969s) [noise=true , address=45.91.116.11:4117 , piece_id=HDFR4DD6YHNB7Q7GSXSHALBSHVOR35U4DQZHFV63WQ5G56C2TMVQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.057722708 (509h25m36.928805753s) [noise=true , address=77.90.31.10:4519 , piece_id=SN7AGZOMRJ5FNDZ7X2UYMPMQPMKVVVYBYI5VO36SZMCUWPE22K6Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.065492483 (509h25m36.936575518s) [noise=true , address=77.90.31.17:4770 , piece_id=YRFRM5NZ66L3GK37HYJ6HTVAGIUP637ME2NQQAJXZDTDXQ6AEJ2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.184083302 (509h25m37.055166337s) [noise=true , address=77.90.31.18:4808 , piece_id=OT6246Q5SFKON4K6CJ6XTJUJEOASJHFKYPOHZXDJ7UNCYEV27G4Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.248613047 (509h25m37.119696082s) [noise=true , address=45.91.116.4:4939 , piece_id=7YU6PZYXKJPPP2IZW2B4CHOPWBSVPBGPKF6JFDRPAUJBXGQQDXEA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.297625265 (509h25m37.1687083s) [noise=true , address=145.79.227.15:4876 , piece_id=LHNJAUZKNGLTZNZ3XHHPYMXBD6IZZDUOPOTMHEH3WTOTAQ7JDSHQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:37.831319745 (509h25m37.7024028s) [noise=true , address=145.79.227.42:4121 , piece_id=6SWFWRHUXQGWFOA3G7W6CGN4DLRDLU7IO6RP4PPFTY5XZO7WYCKA, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078143281 (509h25m37.949226326s) [noise=true , address=145.79.227.25:4618 , piece_id=6PIUKKG2GRU3UZZMACZEOIX3B7K2ERO5YEEJ6IM53NZHYEDNRF2A, seg_pos={0 0}, upload=447028]: Done uploading piece. err: <nil> Dec 26 20:54:38.078160894 (509h25m37.949243939s) [seg_pos={0 0}, upload=447028]: Segment reached optimal threshold of 13 pieces. Dec 26 20:54:38.078391213 (509h25m37.949474248s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 26 20:54:38.078401642 (509h25m37.949484707s) [noise=true , address=145.79.227.40:4051 , piece_id=URYCFS77V3EY4ZOUUB7MMAVNQK6ARPT5J4564ODS5JSIXCHSM54Q, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078529115 (509h25m37.94961215s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39960->145.79.227.51:4410: use of closed network connection Dec 26 20:54:38.078536780 (509h25m37.949619815s) [noise=true , address=145.79.227.51:4410 , piece_id=DUTLZK7BQYJTO45TTLLJW6ESKA7EIH2PGVILZ66PUDMCWJGUILOQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078574421 (509h25m37.949657456s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41662->145.79.227.16:4909: use of closed network connection Dec 26 20:54:38.078580232 (509h25m37.949663267s) [noise=true , address=145.79.227.16:4909 , piece_id=RENEJF3FIQ4XX43QZHAM3O3XMZL4KQEGYPNYZZUBQP6T6RCTNYRQ, seg_pos={0 0}, upload=447028]: Long tail cancellation due to optimal threshold Dec 26 20:54:38.078595392 (509h25m37.949678427s) [seg_pos={0 0}, upload=447028]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:54:38.078655565 (509h25m37.949738601s) [upload=447028]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:54:38.078659643 (509h25m37.949742688s) [upload=447028]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:54:38.382311130 (509h25m38.253394165s) [upload=447028]: Done uploading segments... Dec 26 20:54:38.382329254 (509h25m38.253412289s) [upload=447028]: Upload finished. err: <nil> Duration: 30.008265202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.X....X..XXXX.X...X.X...XX.XXX...X.XX..XXX...............] duration: 6.062260802s trace: https://jaeger.dev.storj.io/trace/74f1a6d8b136b0e5 FAIL (2025-12-26 20:53:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 20:44:02.069091612 (509h15m1.940174667s) [upload=446866]: Starting upload Dec 26 20:44:02.069184279 (509h15m1.940267324s) [upload=446866]: Done starting upload Dec 26 20:44:02.069438933 (509h15m1.940521978s) [upload=446866]: Uploading segments... Dec 26 20:44:02.069499910 (509h15m1.940582955s) [upload=446866]: Got next segment. Inline: false Dec 26 20:44:02.069543673 (509h15m1.940626718s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 20:44:02.069550346 (509h15m1.940633391s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 20:44:02.316250797 (509h15m2.187333852s) [seg_pos={0 0}, upload=446866]: Begin upload segment... Dec 26 20:44:02.316324246 (509h15m2.187407291s) [seg_pos={0 0}, upload=446866]: Done begin upload segment. Dec 26 20:44:02.316425269 (509h15m2.187508314s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.316450788 (509h15m2.187533833s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.317366281 (509h15m2.188449326s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318010909 (509h15m2.189093964s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.318627423 (509h15m2.189710468s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319089092 (509h15m2.190172187s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.319554508 (509h15m2.190637563s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320035614 (509h15m2.191118659s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.320661836 (509h15m2.191744891s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.321837345 (509h15m2.19292039s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.323658823 (509h15m2.194741868s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.325210157 (509h15m2.196293202s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.327058527 (509h15m2.198141582s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.329025442 (509h15m2.200108487s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.330861668 (509h15m2.201944714s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:02.332824476 (509h15m2.203907521s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Uploading piece... Dec 26 20:44:03.938692604 (509h15m3.809775649s) [upload=446866]: Next returned nil segment Dec 26 20:44:03.938707583 (509h15m3.809790618s) [upload=446866]: Waiting for error group managing segments... Dec 26 20:44:04.321157135 (509h15m4.19224017s) [noise=true , address=145.79.227.37:4566 , piece_id=IC7XDR4DUSXOUE2ABVMXN6CB3V3RJSP4HBSDBWUDSUFMGG6ZNBMA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.328171672 (509h15m4.199254717s) [noise=true , address=145.79.227.2:4113 , piece_id=ZNKZZAG3U4MUE4NDHAT2DGNVV4HZ5IBPJ5HMC6O2SEOOQZC35O5A, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.329573030 (509h15m4.200656086s) [noise=true , address=145.79.227.38:4977 , piece_id=POGJE7U5XRSJVP6MLVGFD7D2FU2EAJYCJY76HLNGOMDTGGWZ2XUQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.333517040 (509h15m4.204600085s) [noise=true , address=145.79.227.45:4230 , piece_id=3OSVQMQVHPSYJZZI7FDV4LUCT4C2L2CUKSHHE2ZYU63PAOY4BJ6Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.338016909 (509h15m4.209099965s) [noise=true , address=145.79.227.25:4618 , piece_id=3FFXDORLSR2JV6Q5J5JZMH2YBCMII2YK5NVM2RLP233FDTPSIRIA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.341302986 (509h15m4.212386031s) [noise=true , address=145.79.227.48:4302 , piece_id=KRA3WSK7C4V2SFHZ3XWLPKTY2VR64VQY7BGWSGJPQ2HOE4RJWYBQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.353504190 (509h15m4.224587235s) [noise=true , address=145.79.227.51:4410 , piece_id=UURLBTIJSI72XPKL6CFFLAVJSIFWBVGUNCHV6HGI2EHI4247AJ5Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.410586004 (509h15m4.281669049s) [noise=true , address=145.79.227.6:4259 , piece_id=TPKNTHCM5Q45GIXBP5CSYS5VZDYI2E6BI5LIZ4R7WFFOUIQ2WLTA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.504353794 (509h15m4.375436839s) [noise=true , address=77.90.31.21:4877 , piece_id=XKRO6CSF66LHJOMN5VXE6YQFSFJH3LJPBFBD2GRY4M4ZDT53N24Q, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.530568712 (509h15m4.401651757s) [noise=true , address=45.91.116.8:4045 , piece_id=5F36MPLIBSVYNQ43D2C3AC2WPDCML4DH3V4PHYWJGDDJLVGDQ7CQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.580223113 (509h15m4.451306158s) [noise=true , address=77.90.31.14:4662 , piece_id=7F4YBHYQNBTZLBFT2C4CZM3LUFPXWXKR2LO4Y5I7RGLPTNDBXFZQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.594809889 (509h15m4.465892924s) [noise=true , address=45.91.116.4:4937 , piece_id=345L5RAFMIXUOZJA3TNG2JESV6GRU7X34DA6QVREGAA3HWPH6UOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632213152 (509h15m4.503296197s) [noise=true , address=77.90.31.10:4517 , piece_id=NYJLMTJ3AG42KUHNB4AKORVTNU2WU7LIFJH6UUYCQZNCD4S4UPOA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: <nil> Dec 26 20:44:04.632239601 (509h15m4.503322646s) [seg_pos={0 0}, upload=446866]: Segment reached optimal threshold of 13 pieces. Dec 26 20:44:04.632560503 (509h15m4.503643568s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632569760 (509h15m4.503652805s) [noise=true , address=145.79.227.44:4193 , piece_id=3XBE4EZAYMK5XJWTG3SNIYXIXMVT45OVYEHSRY77TYEBXBG2INQQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.632965033 (509h15m4.504048078s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.632975532 (509h15m4.504058577s) [noise=true , address=45.91.116.11:4119 , piece_id=M65BBQLLTEE5YCIG5R5JSR3757NZSCLT5GKJ7MT62FCSGZWPJZHA, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633177667 (509h15m4.504260733s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 26 20:44:04.633186395 (509h15m4.50426944s) [noise=true , address=45.91.116.13:4154 , piece_id=3AFE67WFFWYUQGHWVPRA7HBXNJONPVFYJ2T5OP2NNP7ARJ2XFXYQ, seg_pos={0 0}, upload=446866]: Long tail cancellation due to optimal threshold Dec 26 20:44:04.633239726 (509h15m4.504322781s) [seg_pos={0 0}, upload=446866]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 20:44:04.633594000 (509h15m4.504677045s) [upload=446866]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 20:44:04.633603499 (509h15m4.504686544s) [upload=446866]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 20:44:04.924053387 (509h15m4.795136442s) [upload=446866]: Done uploading segments... Dec 26 20:44:04.924078444 (509h15m4.795161489s) [upload=446866]: Upload finished. err: <nil> Duration: 6.062294415s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)