Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 35.142720974s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:59:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 35.142741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XX.X.....XX..............X.X.......X.X..........XX.X........] duration: 6.106268396s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 6.106296028s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration: 30.00206823s trace: https://jaeger.dev.storj.io/trace/1aad3210b0015564 FAIL (2025-12-28 12:59:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002079162s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 28.027503389s trace: https://jaeger.dev.storj.io/trace/41e47b0835c5fd15 FAIL (2025-12-28 12:59:25 UTC) orbiter warning: context deadline exceeded Duration: 28.027527306s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 48.008263505s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:59:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.008287541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.XX.X.....XX..............X.X.......X.X..........XX.X.......] duration: 6.181825352s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:59:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 6.181854227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [...X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..XX.X.X] duration: 5.001221955s trace: https://jaeger.dev.storj.io/trace/60f817a2ad36aa5 FAIL (2025-12-28 12:59:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001251391s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 21.197384132s trace: https://jaeger.dev.storj.io/trace/41e47b0835c5fd15 FAIL (2025-12-28 12:58:50 UTC) orbiter warning: context deadline exceeded Duration: 21.197409922s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XX.X.....XX..............X.X.......X.X..........XX.X......] duration: 5.958186744s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:58:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 5.958201764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [....X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..XX.X.] duration: 5.000986457s trace: https://jaeger.dev.storj.io/trace/5f111c864add0687 FAIL (2025-12-28 12:58:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001016303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 48.918653525s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:58:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.918833136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XX.X.....XX..............X.X.......X.X..........XX.X.....] duration: 5.737845378s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:57:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 5.737871107s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 29.013031569s trace: https://jaeger.dev.storj.io/trace/41e47b0835c5fd15 FAIL (2025-12-28 12:57:48 UTC) orbiter warning: context deadline exceeded Duration: 29.01307497s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 28.998735256s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:57:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 28.998761506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.....X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..XX.X] duration: 5.00080521s trace: https://jaeger.dev.storj.io/trace/5f111c864add0687 FAIL (2025-12-28 12:57:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000824406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....XX.X.....XX..............X.X.......X.X..........XX.X....] duration: 6.604474475s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 6.604521967s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 18.211663148s trace: https://jaeger.dev.storj.io/trace/41e47b0835c5fd15 FAIL (2025-12-28 12:56:53 UTC) orbiter warning: context deadline exceeded Duration: 18.211689839s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 49.867556424s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:56:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 49.867578314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..XX.] duration: 3.481789341s trace: https://jaeger.dev.storj.io/trace/6ea04ef88546402a FAIL (2025-12-28 12:56:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.481809999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....XX.X.....XX..............X.X.......X.X..........XX.X...] duration: 5.868492403s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:56:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 5.868514034s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.112471307s trace: https://jaeger.dev.storj.io/trace/41e47b0835c5fd15 FAIL (2025-12-28 12:56:15 UTC) orbiter warning: context deadline exceeded Duration: 27.112516141s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....XX.X.....XX..............X.X.......X.X..........XX.X..] duration: 5.936845589s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:56:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 5.936872509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 48.968478885s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:55:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 48.968505013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00100068s trace: https://jaeger.dev.storj.io/trace/41e47b0835c5fd15 FAIL (2025-12-28 12:55:43 UTC) orbiter warning: context deadline exceeded Duration: 30.001028403s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..XX] duration: 5.000905131s trace: https://jaeger.dev.storj.io/trace/6ea04ef88546402a FAIL (2025-12-28 12:55:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000929328s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X....XX.X.....XX..............X.X.......X.X..........XX.X.] duration: 22.735028052s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:55:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 22.73504271s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..X] duration: 5.00120392s trace: https://jaeger.dev.storj.io/trace/55a3ec27a50470f5 FAIL (2025-12-28 12:55:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001230061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX.X....XX.X.....XX..............X.X.......X.X..........XX.X] duration: 30.002844978s trace: https://jaeger.dev.storj.io/trace/18e04a46fea0e467 FAIL (2025-12-28 12:54:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q) context deadline exceeded ======LOGS====== Dec 28 12:54:28.812776027 (549h25m28.683859072s) [upload=481726]: Starting upload Dec 28 12:54:28.812880907 (549h25m28.683963952s) [upload=481726]: Done starting upload Dec 28 12:54:28.812988782 (549h25m28.684071837s) [upload=481726]: Uploading segments... Dec 28 12:54:28.813142034 (549h25m28.684225089s) [upload=481726]: Got next segment. Inline: false Dec 28 12:54:28.813185447 (549h25m28.684268482s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:54:28.813191398 (549h25m28.684274443s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:54:29.086586758 (549h25m28.957669824s) [seg_pos={0 0}, upload=481726]: Begin upload segment... Dec 28 12:54:29.086643836 (549h25m28.957726902s) [seg_pos={0 0}, upload=481726]: Done begin upload segment. Dec 28 12:54:29.086679414 (549h25m28.957762469s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.086777471 (549h25m28.957860516s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087387092 (549h25m28.958470127s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.087444641 (549h25m28.958527686s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.088135757 (549h25m28.959218803s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089322978 (549h25m28.960406023s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.089653628 (549h25m28.960736703s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.097368850 (549h25m28.968451895s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.098486137 (549h25m28.969569212s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.100053011 (549h25m28.971136056s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.104372125 (549h25m28.97545517s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.108456011 (549h25m28.979539066s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.110176236 (549h25m28.981259271s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.112906035 (549h25m28.98398908s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.115911447 (549h25m28.986994482s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:29.121590060 (549h25m28.992673105s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Uploading piece... Dec 28 12:54:30.754521084 (549h25m30.625604119s) [upload=481726]: Next returned nil segment Dec 28 12:54:30.754533006 (549h25m30.625616041s) [upload=481726]: Waiting for error group managing segments... Dec 28 12:54:31.089301622 (549h25m30.960384657s) [noise=true , address=145.79.227.6:4257 , piece_id=BBHRRYB7BFMNR7DWRDOO2RDPJUDBURPSIERDYZW6N5P2AT3F6KQQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.089636079 (549h25m30.960719124s) [noise=true , address=145.79.227.47:4265 , piece_id=3VPJQYVTQCUKWPOCDWB2P7DA42RVA26MMLA5AMSGZC4SJO3E7XYQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119276639 (549h25m30.990359674s) [noise=true , address=145.79.227.18:4007 , piece_id=BQYDXZWTGHXTZ6JOAM6FECGDJIZWKXMOQVR2JM2WZIUYQYAG5AWQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.119890418 (549h25m30.990973453s) [noise=true , address=145.79.227.50:4374 , piece_id=THTRI57IZUVBKW3DJDH7K5LAOJKIW3T22HXGX7WQYHUKGNJCPG3A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.120326538 (549h25m30.991409583s) [noise=true , address=145.79.227.16:4912 , piece_id=ACMDS56XLBBFFCZR7UQ7XE3F5THSSBFARSRP45XSFJXM4DGPA4XA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.137720840 (549h25m31.008803875s) [noise=true , address=145.79.227.12:4767 , piece_id=QPQT5NTIULW7EE3MVKRUEZ2NBJEHR4QWMA2PJGLDLCFWEFH6GBBQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.148584687 (549h25m31.019667732s) [noise=true , address=145.79.227.51:4411 , piece_id=QBWYKZWDIUWBZELVQOTLBYWZJJQCNMZQ5C4KOMYXWCVJ46P42WOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.151747069 (549h25m31.022830114s) [noise=true , address=145.79.227.2:4113 , piece_id=Z6TBRR27RUYHZ6HOEIOVOFANP6JIEPRY6TCIQIYEQEBWMHNJJ6CQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.155166099 (549h25m31.026249134s) [noise=true , address=145.79.227.32:4836 , piece_id=ITLLTNCQNZBI73ZINXWEYJ6G6QFASDGQFXRTY4QQ4HLWTHRSAV2A, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.164963825 (549h25m31.03604686s) [noise=true , address=145.79.227.5:4223 , piece_id=FA26AB44X5QO2PTSC5OW7T4DIRPVAHDC4SBLTWVJSHJPJYUUET2Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.293381845 (549h25m31.16446491s) [noise=true , address=45.91.116.12:4228 , piece_id=MGCS3UFDQNO4L7MGQXFNOKE2SATNK4KMONLPFN36NUXBVFW542VQ, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.306889456 (549h25m31.177972491s) [noise=true , address=77.90.31.15:4700 , piece_id=HZYD3M6GCWFFVGOBVE3DDRTSUAHRAHSJW6KU5KN5WQG7IOI3OD4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315498048 (549h25m31.186581083s) [noise=true , address=77.90.31.12:4590 , piece_id=4HY5AVL2ZBUAOSIVSMWOTAQOXJXFAXSNVUIPQQYVZBFIUVHO67VA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: <nil> Dec 28 12:54:31.315517285 (549h25m31.18660032s) [seg_pos={0 0}, upload=481726]: Segment reached optimal threshold of 13 pieces. Dec 28 12:54:31.315696115 (549h25m31.18677916s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315702147 (549h25m31.186785182s) [noise=true , address=77.90.31.19:4842 , piece_id=F7JA5JLS5CUTQFAS7WZA3C5HDQJCAYCOBS4PPMRHOR6KO4VYDLOA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315830141 (549h25m31.186913186s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315838456 (549h25m31.186921491s) [noise=true , address=77.90.31.13:4629 , piece_id=SH6P77Z7FYXYON6QU2DF3DF5MTTP5K4G3GAOJYX5XZRUCCFZ3L4Q, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315856620 (549h25m31.186939655s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 28 12:54:31.315864345 (549h25m31.18694739s) [noise=true , address=77.90.31.10:4517 , piece_id=MHI6TRY2DRWERFPR5Y4C2UHGEYL2CB3IM27FEFCFUVACZKPTRMRA, seg_pos={0 0}, upload=481726]: Long tail cancellation due to optimal threshold Dec 28 12:54:31.315879084 (549h25m31.186962129s) [seg_pos={0 0}, upload=481726]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:54:31.315955068 (549h25m31.187038113s) [upload=481726]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:54:31.315959847 (549h25m31.187042882s) [upload=481726]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:54:31.610481636 (549h25m31.481564681s) [upload=481726]: Done uploading segments... Dec 28 12:54:31.610500963 (549h25m31.481584008s) [upload=481726]: Upload finished. err: <nil> Duration: 30.002878753s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X..] duration: 3.495840796s trace: https://jaeger.dev.storj.io/trace/74811d0fbbb66fc2 FAIL (2025-12-28 12:54:31 UTC) orbiter warning: context deadline exceeded Duration: 3.4958586s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 15.181197039s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:54:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 15.181228499s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X.] duration: 3.538668449s trace: https://jaeger.dev.storj.io/trace/74811d0fbbb66fc2 FAIL (2025-12-28 12:53:45 UTC) orbiter warning: context deadline exceeded Duration: 3.538695139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.XX.X....XX.X.....XX..............X.X.......X.X..........XX.] duration: 6.704826433s trace: https://jaeger.dev.storj.io/trace/1f46461679f8010b FAIL (2025-12-28 12:53:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6MXT57GCV3A5TRH22C7BD2LCVFR7337PARXHZK2SFCZGBRXQK2WA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DIQT3DCJ364J4WTYNVDWW7YNJCFQFXWUPUVPZPJXVY2P5BKGDBQA) context deadline exceeded ======LOGS====== 8 12:52:51.875659836 (549h23m51.746742881s) [upload=481705]: Starting upload Dec 28 12:52:51.875744487 (549h23m51.746827522s) [upload=481705]: Done starting upload Dec 28 12:52:51.875828837 (549h23m51.746911892s) [upload=481705]: Uploading segments... Dec 28 12:52:51.876023268 (549h23m51.747106303s) [upload=481705]: Got next segment. Inline: false Dec 28 12:52:51.876084094 (549h23m51.747167149s) [upload=481705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:52:51.876090376 (549h23m51.747173431s) [upload=481705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:52:52.121651455 (549h23m51.99273449s) [seg_pos={0 0}, upload=481705]: Begin upload segment... Dec 28 12:52:52.121714035 (549h23m51.99279708s) [seg_pos={0 0}, upload=481705]: Done begin upload segment. Dec 28 12:52:52.121768798 (549h23m51.992851833s) [noise=true , address=45.91.116.12:4227 , piece_id=DIQT3DCJ364J4WTYNVDWW7YNJCFQFXWUPUVPZPJXVY2P5BKGDBQA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.121792714 (549h23m51.992875759s) [noise=true , address=145.79.227.4:4186 , piece_id=QDJ2VMZKTULEL6TTZE6PF2ZQKA5I7RN36GHY4Q5DQ4KRWOVEQCAQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.122686156 (549h23m51.993769201s) [noise=true , address=145.79.227.37:4567 , piece_id=5VLWC56F52R3QQVG2KI4DHKVV7B4ZPPHLOTNXUYM2LIKXMB4POCQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.123014150 (549h23m51.994097185s) [noise=true , address=45.91.116.5:4298 , piece_id=WA7LD4MEWUYZTL2EYTECI7SB6AK6TQRZU2B6NSKPJFPFGKCXU4KA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.123443788 (549h23m51.994526823s) [noise=true , address=145.79.227.36:4518 , piece_id=SU27OB6FH5CNQG2HVBIPVOVYST2WGG75HYFIBYLIYYYPK5ZMVN2Q, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.123973547 (549h23m51.995056592s) [noise=true , address=45.91.116.13:4154 , piece_id=EKWTDH5Z3MPKOVMDVXDGVYRCTRMS2J7E44JZFX4NKIAOR2G2PXGQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.124393797 (549h23m51.995476842s) [noise=true , address=45.91.116.14:4191 , piece_id=BRB2O44JYB3Z3MXYIVE3KLZ7GQAFQPG55W6GK3BTPELEL6IR2YYQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.124736931 (549h23m51.995819966s) [noise=true , address=145.79.227.45:4231 , piece_id=COT25MWJTPYZCV3L5PVFVACVEBSSVUG47LEQONL6Q7NWZ4JGBA7A, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.125102316 (549h23m51.996185351s) [noise=true , address=145.79.227.5:4221 , piece_id=LUIV6J6HV3IBBMLWWJDN3QZM76WKOVROLDL7GGMQM63DEU6O7K5Q, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.125449417 (549h23m51.996532452s) [noise=true , address=145.79.227.13:4802 , piece_id=HE5NJQVSFVXWGG7AFIVDLVUZGSEIZYLCWFV3ZUFOJKNHJNFOBPUA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.126544333 (549h23m51.997627378s) [noise=true , address=145.79.227.9:4367 , piece_id=YZPPWZLARESTVPSLKBI726LXO2QP4PPVULCSEPV2PXBWT5UEVLHQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.127250076 (549h23m51.998333121s) [noise=true , address=45.91.116.6:4974 , piece_id=XWJKLT4ZNKB42AJL3IXCGLIYDQ2T4ET55YAAIGBJQXNS5YDZIFXQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.127764346 (549h23m51.998847391s) [noise=true , address=145.79.227.29:4942 , piece_id=J3TSLGA4G5GGDXLWXPQVYEAYRQPYXOOT7AP6KM26V3XDOPRJWT5A, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.128276290 (549h23m51.999359345s) [noise=true , address=45.91.116.10:4082 , piece_id=PHA6DW2G6QYDJ6WFB6EJSAPMCGQZLOQTY7APJQDP3K3LP2Z6GPVA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.128813774 (549h23m51.999896819s) [noise=true , address=145.79.227.38:4978 , piece_id=J6O2PEJ6SIF747HGOU2HGRXV6SVLUVZWKZVBKFPYXLNN57QLTYMQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.129341559 (549h23m52.000424604s) [noise=true , address=45.91.116.9:4263 , piece_id=6MXT57GCV3A5TRH22C7BD2LCVFR7337PARXHZK2SFCZGBRXQK2WA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:53.856200208 (549h23m53.727283253s) [upload=481705]: Next returned nil segment Dec 28 12:52:53.856215947 (549h23m53.727298992s) [upload=481705]: Waiting for error group managing segments... Dec 28 12:52:54.132770023 (549h23m54.003853058s) [noise=true , address=145.79.227.5:4221 , piece_id=LUIV6J6HV3IBBMLWWJDN3QZM76WKOVROLDL7GGMQM63DEU6O7K5Q, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.140585004 (549h23m54.011668039s) [noise=true , address=145.79.227.37:4567 , piece_id=5VLWC56F52R3QQVG2KI4DHKVV7B4ZPPHLOTNXUYM2LIKXMB4POCQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.140869976 (549h23m54.011953021s) [noise=true , address=145.79.227.45:4231 , piece_id=COT25MWJTPYZCV3L5PVFVACVEBSSVUG47LEQONL6Q7NWZ4JGBA7A, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.146930096 (549h23m54.018013141s) [noise=true , address=145.79.227.29:4942 , piece_id=J3TSLGA4G5GGDXLWXPQVYEAYRQPYXOOT7AP6KM26V3XDOPRJWT5A, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.150960931 (549h23m54.022043966s) [noise=true , address=145.79.227.38:4978 , piece_id=J6O2PEJ6SIF747HGOU2HGRXV6SVLUVZWKZVBKFPYXLNN57QLTYMQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.158735255 (549h23m54.02981829s) [noise=true , address=145.79.227.9:4367 , piece_id=YZPPWZLARESTVPSLKBI726LXO2QP4PPVULCSEPV2PXBWT5UEVLHQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.158985250 (549h23m54.030068296s) [noise=true , address=145.79.227.36:4518 , piece_id=SU27OB6FH5CNQG2HVBIPVOVYST2WGG75HYFIBYLIYYYPK5ZMVN2Q, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.160201837 (549h23m54.031284872s) [noise=true , address=145.79.227.13:4802 , piece_id=HE5NJQVSFVXWGG7AFIVDLVUZGSEIZYLCWFV3ZUFOJKNHJNFOBPUA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.175508364 (549h23m54.046591409s) [noise=true , address=145.79.227.4:4186 , piece_id=QDJ2VMZKTULEL6TTZE6PF2ZQKA5I7RN36GHY4Q5DQ4KRWOVEQCAQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.414996610 (549h23m54.286079655s) [noise=true , address=45.91.116.10:4082 , piece_id=PHA6DW2G6QYDJ6WFB6EJSAPMCGQZLOQTY7APJQDP3K3LP2Z6GPVA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.479518069 (549h23m54.350601115s) [noise=true , address=45.91.116.12:4227 , piece_id=DIQT3DCJ364J4WTYNVDWW7YNJCFQFXWUPUVPZPJXVY2P5BKGDBQA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.498519420 (549h23m54.369602466s) [noise=true , address=45.91.116.14:4191 , piece_id=BRB2O44JYB3Z3MXYIVE3KLZ7GQAFQPG55W6GK3BTPELEL6IR2YYQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.508504033 (549h23m54.379587068s) [noise=true , address=45.91.116.9:4263 , piece_id=6MXT57GCV3A5TRH22C7BD2LCVFR7337PARXHZK2SFCZGBRXQK2WA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.508529071 (549h23m54.379612116s) [seg_pos={0 0}, upload=481705]: Segment reached optimal threshold of 13 pieces. Dec 28 12:52:54.508763448 (549h23m54.379846513s) [noise=true , address=45.91.116.6:4974 , piece_id=XWJKLT4ZNKB42AJL3IXCGLIYDQ2T4ET55YAAIGBJQXNS5YDZIFXQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:53648->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:53648->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53648->45.91.116.6:4974: use of closed network connection Dec 28 12:52:54.508773206 (549h23m54.379856261s) [noise=true , address=45.91.116.6:4974 , piece_id=XWJKLT4ZNKB42AJL3IXCGLIYDQ2T4ET55YAAIGBJQXNS5YDZIFXQ, seg_pos={0 0}, upload=481705]: Long tail cancellation due to optimal threshold Dec 28 12:52:54.508783716 (549h23m54.379866761s) [noise=true , address=45.91.116.5:4298 , piece_id=WA7LD4MEWUYZTL2EYTECI7SB6AK6TQRZU2B6NSKPJFPFGKCXU4KA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 12:52:54.508792342 (549h23m54.379875377s) [noise=true , address=45.91.116.5:4298 , piece_id=WA7LD4MEWUYZTL2EYTECI7SB6AK6TQRZU2B6NSKPJFPFGKCXU4KA, seg_pos={0 0}, upload=481705]: Long tail cancellation due to optimal threshold Dec 28 12:52:54.509033721 (549h23m54.380116767s) [noise=true , address=45.91.116.13:4154 , piece_id=EKWTDH5Z3MPKOVMDVXDGVYRCTRMS2J7E44JZFX4NKIAOR2G2PXGQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 12:52:54.509042709 (549h23m54.380125744s) [noise=true , address=45.91.116.13:4154 , piece_id=EKWTDH5Z3MPKOVMDVXDGVYRCTRMS2J7E44JZFX4NKIAOR2G2PXGQ, seg_pos={0 0}, upload=481705]: Long tail cancellation due to optimal threshold Dec 28 12:52:54.509062597 (549h23m54.380145632s) [seg_pos={0 0}, upload=481705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:52:54.509143350 (549h23m54.380226385s) [upload=481705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:52:54.509152348 (549h23m54.380235383s) [upload=481705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:52:54.803913863 (549h23m54.674996898s) [upload=481705]: Done uploading segments... Dec 28 12:52:54.803938530 (549h23m54.675021576s) [upload=481705]: Upload finished. err: <nil> Duration: 6.704852812s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 30.779830018s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 30.779871488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XX.X....XX.X.....XX..............X.X.......X.X..........XX] duration: 30.002790157s trace: https://jaeger.dev.storj.io/trace/1f46461679f8010b FAIL (2025-12-28 12:53:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 6MXT57GCV3A5TRH22C7BD2LCVFR7337PARXHZK2SFCZGBRXQK2WA) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: DIQT3DCJ364J4WTYNVDWW7YNJCFQFXWUPUVPZPJXVY2P5BKGDBQA) context deadline exceeded ======LOGS====== 8 12:52:51.875659836 (549h23m51.746742881s) [upload=481705]: Starting upload Dec 28 12:52:51.875744487 (549h23m51.746827522s) [upload=481705]: Done starting upload Dec 28 12:52:51.875828837 (549h23m51.746911892s) [upload=481705]: Uploading segments... Dec 28 12:52:51.876023268 (549h23m51.747106303s) [upload=481705]: Got next segment. Inline: false Dec 28 12:52:51.876084094 (549h23m51.747167149s) [upload=481705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:52:51.876090376 (549h23m51.747173431s) [upload=481705]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:52:52.121651455 (549h23m51.99273449s) [seg_pos={0 0}, upload=481705]: Begin upload segment... Dec 28 12:52:52.121714035 (549h23m51.99279708s) [seg_pos={0 0}, upload=481705]: Done begin upload segment. Dec 28 12:52:52.121768798 (549h23m51.992851833s) [noise=true , address=45.91.116.12:4227 , piece_id=DIQT3DCJ364J4WTYNVDWW7YNJCFQFXWUPUVPZPJXVY2P5BKGDBQA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.121792714 (549h23m51.992875759s) [noise=true , address=145.79.227.4:4186 , piece_id=QDJ2VMZKTULEL6TTZE6PF2ZQKA5I7RN36GHY4Q5DQ4KRWOVEQCAQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.122686156 (549h23m51.993769201s) [noise=true , address=145.79.227.37:4567 , piece_id=5VLWC56F52R3QQVG2KI4DHKVV7B4ZPPHLOTNXUYM2LIKXMB4POCQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.123014150 (549h23m51.994097185s) [noise=true , address=45.91.116.5:4298 , piece_id=WA7LD4MEWUYZTL2EYTECI7SB6AK6TQRZU2B6NSKPJFPFGKCXU4KA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.123443788 (549h23m51.994526823s) [noise=true , address=145.79.227.36:4518 , piece_id=SU27OB6FH5CNQG2HVBIPVOVYST2WGG75HYFIBYLIYYYPK5ZMVN2Q, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.123973547 (549h23m51.995056592s) [noise=true , address=45.91.116.13:4154 , piece_id=EKWTDH5Z3MPKOVMDVXDGVYRCTRMS2J7E44JZFX4NKIAOR2G2PXGQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.124393797 (549h23m51.995476842s) [noise=true , address=45.91.116.14:4191 , piece_id=BRB2O44JYB3Z3MXYIVE3KLZ7GQAFQPG55W6GK3BTPELEL6IR2YYQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.124736931 (549h23m51.995819966s) [noise=true , address=145.79.227.45:4231 , piece_id=COT25MWJTPYZCV3L5PVFVACVEBSSVUG47LEQONL6Q7NWZ4JGBA7A, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.125102316 (549h23m51.996185351s) [noise=true , address=145.79.227.5:4221 , piece_id=LUIV6J6HV3IBBMLWWJDN3QZM76WKOVROLDL7GGMQM63DEU6O7K5Q, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.125449417 (549h23m51.996532452s) [noise=true , address=145.79.227.13:4802 , piece_id=HE5NJQVSFVXWGG7AFIVDLVUZGSEIZYLCWFV3ZUFOJKNHJNFOBPUA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.126544333 (549h23m51.997627378s) [noise=true , address=145.79.227.9:4367 , piece_id=YZPPWZLARESTVPSLKBI726LXO2QP4PPVULCSEPV2PXBWT5UEVLHQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.127250076 (549h23m51.998333121s) [noise=true , address=45.91.116.6:4974 , piece_id=XWJKLT4ZNKB42AJL3IXCGLIYDQ2T4ET55YAAIGBJQXNS5YDZIFXQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.127764346 (549h23m51.998847391s) [noise=true , address=145.79.227.29:4942 , piece_id=J3TSLGA4G5GGDXLWXPQVYEAYRQPYXOOT7AP6KM26V3XDOPRJWT5A, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.128276290 (549h23m51.999359345s) [noise=true , address=45.91.116.10:4082 , piece_id=PHA6DW2G6QYDJ6WFB6EJSAPMCGQZLOQTY7APJQDP3K3LP2Z6GPVA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.128813774 (549h23m51.999896819s) [noise=true , address=145.79.227.38:4978 , piece_id=J6O2PEJ6SIF747HGOU2HGRXV6SVLUVZWKZVBKFPYXLNN57QLTYMQ, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:52.129341559 (549h23m52.000424604s) [noise=true , address=45.91.116.9:4263 , piece_id=6MXT57GCV3A5TRH22C7BD2LCVFR7337PARXHZK2SFCZGBRXQK2WA, seg_pos={0 0}, upload=481705]: Uploading piece... Dec 28 12:52:53.856200208 (549h23m53.727283253s) [upload=481705]: Next returned nil segment Dec 28 12:52:53.856215947 (549h23m53.727298992s) [upload=481705]: Waiting for error group managing segments... Dec 28 12:52:54.132770023 (549h23m54.003853058s) [noise=true , address=145.79.227.5:4221 , piece_id=LUIV6J6HV3IBBMLWWJDN3QZM76WKOVROLDL7GGMQM63DEU6O7K5Q, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.140585004 (549h23m54.011668039s) [noise=true , address=145.79.227.37:4567 , piece_id=5VLWC56F52R3QQVG2KI4DHKVV7B4ZPPHLOTNXUYM2LIKXMB4POCQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.140869976 (549h23m54.011953021s) [noise=true , address=145.79.227.45:4231 , piece_id=COT25MWJTPYZCV3L5PVFVACVEBSSVUG47LEQONL6Q7NWZ4JGBA7A, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.146930096 (549h23m54.018013141s) [noise=true , address=145.79.227.29:4942 , piece_id=J3TSLGA4G5GGDXLWXPQVYEAYRQPYXOOT7AP6KM26V3XDOPRJWT5A, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.150960931 (549h23m54.022043966s) [noise=true , address=145.79.227.38:4978 , piece_id=J6O2PEJ6SIF747HGOU2HGRXV6SVLUVZWKZVBKFPYXLNN57QLTYMQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.158735255 (549h23m54.02981829s) [noise=true , address=145.79.227.9:4367 , piece_id=YZPPWZLARESTVPSLKBI726LXO2QP4PPVULCSEPV2PXBWT5UEVLHQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.158985250 (549h23m54.030068296s) [noise=true , address=145.79.227.36:4518 , piece_id=SU27OB6FH5CNQG2HVBIPVOVYST2WGG75HYFIBYLIYYYPK5ZMVN2Q, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.160201837 (549h23m54.031284872s) [noise=true , address=145.79.227.13:4802 , piece_id=HE5NJQVSFVXWGG7AFIVDLVUZGSEIZYLCWFV3ZUFOJKNHJNFOBPUA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.175508364 (549h23m54.046591409s) [noise=true , address=145.79.227.4:4186 , piece_id=QDJ2VMZKTULEL6TTZE6PF2ZQKA5I7RN36GHY4Q5DQ4KRWOVEQCAQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.414996610 (549h23m54.286079655s) [noise=true , address=45.91.116.10:4082 , piece_id=PHA6DW2G6QYDJ6WFB6EJSAPMCGQZLOQTY7APJQDP3K3LP2Z6GPVA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.479518069 (549h23m54.350601115s) [noise=true , address=45.91.116.12:4227 , piece_id=DIQT3DCJ364J4WTYNVDWW7YNJCFQFXWUPUVPZPJXVY2P5BKGDBQA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.498519420 (549h23m54.369602466s) [noise=true , address=45.91.116.14:4191 , piece_id=BRB2O44JYB3Z3MXYIVE3KLZ7GQAFQPG55W6GK3BTPELEL6IR2YYQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.508504033 (549h23m54.379587068s) [noise=true , address=45.91.116.9:4263 , piece_id=6MXT57GCV3A5TRH22C7BD2LCVFR7337PARXHZK2SFCZGBRXQK2WA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: <nil> Dec 28 12:52:54.508529071 (549h23m54.379612116s) [seg_pos={0 0}, upload=481705]: Segment reached optimal threshold of 13 pieces. Dec 28 12:52:54.508763448 (549h23m54.379846513s) [noise=true , address=45.91.116.6:4974 , piece_id=XWJKLT4ZNKB42AJL3IXCGLIYDQ2T4ET55YAAIGBJQXNS5YDZIFXQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:53648->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:53648->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53648->45.91.116.6:4974: use of closed network connection Dec 28 12:52:54.508773206 (549h23m54.379856261s) [noise=true , address=45.91.116.6:4974 , piece_id=XWJKLT4ZNKB42AJL3IXCGLIYDQ2T4ET55YAAIGBJQXNS5YDZIFXQ, seg_pos={0 0}, upload=481705]: Long tail cancellation due to optimal threshold Dec 28 12:52:54.508783716 (549h23m54.379866761s) [noise=true , address=45.91.116.5:4298 , piece_id=WA7LD4MEWUYZTL2EYTECI7SB6AK6TQRZU2B6NSKPJFPFGKCXU4KA, seg_pos={0 0}, upload=481705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 12:52:54.508792342 (549h23m54.379875377s) [noise=true , address=45.91.116.5:4298 , piece_id=WA7LD4MEWUYZTL2EYTECI7SB6AK6TQRZU2B6NSKPJFPFGKCXU4KA, seg_pos={0 0}, upload=481705]: Long tail cancellation due to optimal threshold Dec 28 12:52:54.509033721 (549h23m54.380116767s) [noise=true , address=45.91.116.13:4154 , piece_id=EKWTDH5Z3MPKOVMDVXDGVYRCTRMS2J7E44JZFX4NKIAOR2G2PXGQ, seg_pos={0 0}, upload=481705]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 12:52:54.509042709 (549h23m54.380125744s) [noise=true , address=45.91.116.13:4154 , piece_id=EKWTDH5Z3MPKOVMDVXDGVYRCTRMS2J7E44JZFX4NKIAOR2G2PXGQ, seg_pos={0 0}, upload=481705]: Long tail cancellation due to optimal threshold Dec 28 12:52:54.509062597 (549h23m54.380145632s) [seg_pos={0 0}, upload=481705]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:52:54.509143350 (549h23m54.380226385s) [upload=481705]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:52:54.509152348 (549h23m54.380235383s) [upload=481705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:52:54.803913863 (549h23m54.674996898s) [upload=481705]: Done uploading segments... Dec 28 12:52:54.803938530 (549h23m54.675021576s) [upload=481705]: Upload finished. err: <nil> Duration: 30.00281841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..X] duration: 5.001246224s trace: https://jaeger.dev.storj.io/trace/74811d0fbbb66fc2 FAIL (2025-12-28 12:52:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001272804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 52.68186117s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:52:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 52.681891167s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XX.X....XX.X.....XX..............X.X.......X.X..........X] duration: 30.003203588s trace: https://jaeger.dev.storj.io/trace/4beccaedf4135453 FAIL (2025-12-28 12:52:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: 2TWIQRX52II5XYZ7XSUU6MVAAQN6CUZE5B4A5ZH5RLVHTAS6ZKIA) context deadline exceeded; piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: X7Q4Y35EBVJOEPOXBRP2JM7LSMNRJ47W4JQUC76QD7O4HDBYAJPA) context deadline exceeded ======LOGS====== Dec 28 12:51:59.076786770 (549h22m58.947869806s) [upload=481694]: Starting upload Dec 28 12:51:59.076871973 (549h22m58.947955018s) [upload=481694]: Done starting upload Dec 28 12:51:59.077005517 (549h22m58.948088552s) [upload=481694]: Uploading segments... Dec 28 12:51:59.077159681 (549h22m58.948242716s) [upload=481694]: Got next segment. Inline: false Dec 28 12:51:59.077213593 (549h22m58.948296628s) [upload=481694]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:51:59.077218583 (549h22m58.948301618s) [upload=481694]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:51:59.318513820 (549h22m59.189596856s) [seg_pos={0 0}, upload=481694]: Begin upload segment... Dec 28 12:51:59.318564537 (549h22m59.189647582s) [seg_pos={0 0}, upload=481694]: Done begin upload segment. Dec 28 12:51:59.318612998 (549h22m59.189696034s) [noise=true , address=145.79.227.39:4014 , piece_id=MEKSYQOVLK5UBXZDBQUAMITKIUH75EP6MX4R5PU5XGME5NBNEW4A, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.318632015 (549h22m59.18971505s) [noise=true , address=145.79.227.3:4149 , piece_id=QIAREHKPS25NYGGZVRB26CQW7P2RLO44YHS332N3OZ7JZIHIC55A, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.319287343 (549h22m59.190370378s) [noise=true , address=45.91.116.5:4298 , piece_id=X7Q4Y35EBVJOEPOXBRP2JM7LSMNRJ47W4JQUC76QD7O4HDBYAJPA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.319925428 (549h22m59.191008463s) [noise=true , address=145.79.227.11:4729 , piece_id=GXA4WOUIDO2CTXUPSPP3XCYK5GVVLI6TBET2RMQVFUBY6ULHW2DA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.320275494 (549h22m59.191358529s) [noise=true , address=45.91.116.10:4082 , piece_id=PJ6IYLDKOCETRJO4OGWTEFUSYMBMIEBAJRDILAC6XGKJDL6BBXUA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.320589633 (549h22m59.191672678s) [noise=true , address=145.79.227.45:4229 , piece_id=OITA5XXEHSPORTEM2CMXICWJ34RTD2EWNKUCIJ45TOMBQC6VBTLQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.320905103 (549h22m59.191988138s) [noise=true , address=145.79.227.14:4837 , piece_id=JHSCMPMWB2TNPT7TTFA3FW5G3TY5SFI2MQR2SINL2RGK572SBAYA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.321279707 (549h22m59.192362742s) [noise=true , address=45.91.116.7:4012 , piece_id=4YLBAMDQ5GHU7HFNBDV32VQ7XGPUGOD4JVK6CCRM4OQUFTO4LANQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.321676672 (549h22m59.192759707s) [noise=true , address=145.79.227.37:4567 , piece_id=5ET6TMMLSA4SAWJV3YBMDZDNZHG4FO3KCD43I6QTS53UC5GHDLZQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.322155975 (549h22m59.19323902s) [noise=true , address=145.79.227.29:4941 , piece_id=YHYPC3Y32YIXH5TTH7UR3F7YUN7NVAUMFFCLXGH5L6CGSR5OULHQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.322284350 (549h22m59.193367395s) [noise=true , address=145.79.227.48:4302 , piece_id=QQ6ZXJUVPMSDKCR4EKAN6LNTPSZK4OHZBR3TNTLDTLD3RMTA6XEQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.322496504 (549h22m59.193579549s) [noise=true , address=145.79.227.6:4257 , piece_id=JWI6GHXBLW7KEGZXDL4RDUD3SL2HKEMJNZYCCT7YJWHZOSVKXBPQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.322685503 (549h22m59.193768538s) [noise=true , address=145.79.227.51:4410 , piece_id=WN5WL7FVVB2KJHSZXF4XOU7P6RYM7E2YWYFF7EFAP4E4AQ4LE7BA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.322786576 (549h22m59.193869611s) [noise=true , address=145.79.227.20:4439 , piece_id=W6DZHCNVGMOUC6C5NW3QSVCDYYCH556GE355GPIRTIK2TABQXGDA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.322995534 (549h22m59.194078579s) [noise=true , address=77.90.31.11:4555 , piece_id=7HJHBP3DMQUW4EPHKW5DGG25HOBCV6EPNRGXV4VM6CV7L3XXE2YQ, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:51:59.323130871 (549h22m59.194213917s) [noise=true , address=77.90.31.13:4627 , piece_id=2TWIQRX52II5XYZ7XSUU6MVAAQN6CUZE5B4A5ZH5RLVHTAS6ZKIA, seg_pos={0 0}, upload=481694]: Uploading piece... Dec 28 12:52:01.028906907 (549h23m0.899989942s) [upload=481694]: Next returned nil segment Dec 28 12:52:01.028921985 (549h23m0.90000502s) [upload=481694]: Waiting for error group managing segments... Dec 28 12:52:01.314363785 (549h23m1.18544682s) [noise=true , address=145.79.227.6:4257 , piece_id=JWI6GHXBLW7KEGZXDL4RDUD3SL2HKEMJNZYCCT7YJWHZOSVKXBPQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.327177585 (549h23m1.19826063s) [noise=true , address=145.79.227.51:4410 , piece_id=WN5WL7FVVB2KJHSZXF4XOU7P6RYM7E2YWYFF7EFAP4E4AQ4LE7BA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.327641598 (549h23m1.198724643s) [noise=true , address=145.79.227.14:4837 , piece_id=JHSCMPMWB2TNPT7TTFA3FW5G3TY5SFI2MQR2SINL2RGK572SBAYA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.333223187 (549h23m1.204306232s) [noise=true , address=145.79.227.3:4149 , piece_id=QIAREHKPS25NYGGZVRB26CQW7P2RLO44YHS332N3OZ7JZIHIC55A, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.334898567 (549h23m1.205981602s) [noise=true , address=145.79.227.11:4729 , piece_id=GXA4WOUIDO2CTXUPSPP3XCYK5GVVLI6TBET2RMQVFUBY6ULHW2DA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.349089268 (549h23m1.220172313s) [noise=true , address=145.79.227.48:4302 , piece_id=QQ6ZXJUVPMSDKCR4EKAN6LNTPSZK4OHZBR3TNTLDTLD3RMTA6XEQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.352105230 (549h23m1.223188265s) [noise=true , address=145.79.227.20:4439 , piece_id=W6DZHCNVGMOUC6C5NW3QSVCDYYCH556GE355GPIRTIK2TABQXGDA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.352371317 (549h23m1.223454352s) [noise=true , address=145.79.227.45:4229 , piece_id=OITA5XXEHSPORTEM2CMXICWJ34RTD2EWNKUCIJ45TOMBQC6VBTLQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.356067836 (549h23m1.227150881s) [noise=true , address=145.79.227.29:4941 , piece_id=YHYPC3Y32YIXH5TTH7UR3F7YUN7NVAUMFFCLXGH5L6CGSR5OULHQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.373246257 (549h23m1.244329302s) [noise=true , address=145.79.227.39:4014 , piece_id=MEKSYQOVLK5UBXZDBQUAMITKIUH75EP6MX4R5PU5XGME5NBNEW4A, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.373539625 (549h23m1.24462267s) [noise=true , address=145.79.227.37:4567 , piece_id=5ET6TMMLSA4SAWJV3YBMDZDNZHG4FO3KCD43I6QTS53UC5GHDLZQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:01.607554980 (549h23m1.478638025s) [noise=true , address=45.91.116.5:4298 , piece_id=X7Q4Y35EBVJOEPOXBRP2JM7LSMNRJ47W4JQUC76QD7O4HDBYAJPA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:02.348238625 (549h23m2.21932167s) [noise=true , address=77.90.31.13:4627 , piece_id=2TWIQRX52II5XYZ7XSUU6MVAAQN6CUZE5B4A5ZH5RLVHTAS6ZKIA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: <nil> Dec 28 12:52:02.348261109 (549h23m2.219344154s) [seg_pos={0 0}, upload=481694]: Segment reached optimal threshold of 13 pieces. Dec 28 12:52:02.348458925 (549h23m2.21954197s) [noise=true , address=77.90.31.11:4555 , piece_id=7HJHBP3DMQUW4EPHKW5DGG25HOBCV6EPNRGXV4VM6CV7L3XXE2YQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 28 12:52:02.348466189 (549h23m2.219549224s) [noise=true , address=77.90.31.11:4555 , piece_id=7HJHBP3DMQUW4EPHKW5DGG25HOBCV6EPNRGXV4VM6CV7L3XXE2YQ, seg_pos={0 0}, upload=481694]: Long tail cancellation due to optimal threshold Dec 28 12:52:02.348531564 (549h23m2.219614599s) [noise=true , address=45.91.116.10:4082 , piece_id=PJ6IYLDKOCETRJO4OGWTEFUSYMBMIEBAJRDILAC6XGKJDL6BBXUA, seg_pos={0 0}, upload=481694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:56642->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:56642->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56642->45.91.116.10:4082: use of closed network connection Dec 28 12:52:02.348540350 (549h23m2.219623395s) [noise=true , address=45.91.116.10:4082 , piece_id=PJ6IYLDKOCETRJO4OGWTEFUSYMBMIEBAJRDILAC6XGKJDL6BBXUA, seg_pos={0 0}, upload=481694]: Long tail cancellation due to optimal threshold Dec 28 12:52:02.348691598 (549h23m2.219774633s) [noise=true , address=45.91.116.7:4012 , piece_id=4YLBAMDQ5GHU7HFNBDV32VQ7XGPUGOD4JVK6CCRM4OQUFTO4LANQ, seg_pos={0 0}, upload=481694]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 28 12:52:02.348698922 (549h23m2.219781957s) [noise=true , address=45.91.116.7:4012 , piece_id=4YLBAMDQ5GHU7HFNBDV32VQ7XGPUGOD4JVK6CCRM4OQUFTO4LANQ, seg_pos={0 0}, upload=481694]: Long tail cancellation due to optimal threshold Dec 28 12:52:02.348717067 (549h23m2.219800112s) [seg_pos={0 0}, upload=481694]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:52:02.348791568 (549h23m2.219874613s) [upload=481694]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:52:02.348799113 (549h23m2.219882158s) [upload=481694]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:52:02.644988497 (549h23m2.516071552s) [upload=481694]: Done uploading segments... Dec 28 12:52:02.645013555 (549h23m2.51609659s) [upload=481694]: Upload finished. err: <nil> Duration: 30.003225309s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX..] duration: 4.995435238s trace: https://jaeger.dev.storj.io/trace/46e1ec551b406ab3 FAIL (2025-12-28 12:51:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.995464884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 18.726645267s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:51:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 18.726660125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 38.197978333s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:51:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 38.197991258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..XX.X....XX.X.....XX..............X.X.......X.X..........] duration: 5.653522216s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 5.653554497s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX.] duration: 3.554545851s trace: https://jaeger.dev.storj.io/trace/46e1ec551b406ab3 FAIL (2025-12-28 12:51:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.554578002s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..XX.X....XX.X.....XX..............X.X.......X.X.........] duration: 5.936840149s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:50:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 5.936865897s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXXX] duration: 5.001256431s trace: https://jaeger.dev.storj.io/trace/46e1ec551b406ab3 FAIL (2025-12-28 12:50:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001279916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 36.073117594s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 36.07313664s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..XX.X....XX.X.....XX..............X.X.......X.X........] duration: 6.154413507s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 6.154443524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 41.287058353s trace: https://jaeger.dev.storj.io/trace/74ea26f278bb08e5 FAIL (2025-12-28 12:50:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483641569 (548h58m46.354724604s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.483948473 (548h58m46.355031508s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484247723 (548h58m46.355330768s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484538296 (548h58m46.355621331s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.484836064 (548h58m46.355919099s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485109394 (548h58m46.356192429s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485383906 (548h58m46.356466941s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.485655443 (548h58m46.356738478s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486016021 (548h58m46.357099066s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486371417 (548h58m46.357454452s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.486692328 (548h58m46.357775363s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487108631 (548h58m46.358191676s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.487470249 (548h58m46.358553284s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.488579632 (548h58m46.359662667s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:46.489680749 (548h58m46.360763784s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Uploading piece... Dec 28 12:27:47.811519393 (548h58m47.682602439s) [upload=481360]: Next returned nil segment Dec 28 12:27:47.811558428 (548h58m47.682641473s) [upload=481360]: Waiting for error group managing segments... Dec 28 12:27:48.148211973 (548h58m48.019295008s) [noise=true , address=193.149.164.143:4701 , piece_id=G3UYFGVMCPOHDPHD74NBM6UE4QPMX5467DUQYHQDTP5ZJGS4UXZA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.357091706 (548h58m48.228174751s) [noise=true , address=145.79.227.40:4083 , piece_id=73YHG2V5YCLAO5SBD37SDBNGO23T3RSFTUQP6D3VDXZEXQMS7IFA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.374968026 (548h58m48.246051071s) [noise=true , address=145.79.227.6:4266 , piece_id=R6TROOXKQN5YUS2RKWEIPAW7Q2SUIVMTJB2VHZYWOSUS7XP4JUHQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.411613225 (548h58m48.28269626s) [noise=true , address=145.79.227.5:4232 , piece_id=J5TGPGIMU76PSWIKKVW6YMJAHISUZTV4TSB6YAY4NAOZPWXRMS7A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.422667975 (548h58m48.29375101s) [noise=true , address=145.79.227.49:4349 , piece_id=RV2EZ5U5E4VEKO7RHVS6NVSVXOG3AIAQJGGWI64B5HKCXHKFEDIQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.464738914 (548h58m48.335821949s) [noise=true , address=145.79.227.35:4501 , piece_id=ZZPXMNY62KUI6QSZKFQKVTJCNEDQMD4T7UORNUWPFFXBNONL23XA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.573468515 (548h58m48.44455155s) [noise=true , address=77.90.31.18:4835 , piece_id=4RWPHCTAXZHS4IY2JVXCXULYBBQHVQXYDHMPXAU3ATSCADX5HG4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.650573926 (548h58m48.521656961s) [noise=true , address=77.90.31.13:4638 , piece_id=3HA7V4X2PLXHNFODF67MBF5B2355ERPEZKGW36V7XE2Q4NTPOYKQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.651762800 (548h58m48.522845845s) [noise=true , address=77.90.31.17:4794 , piece_id=WHRCXQQ6CGFM2NQGO7AD67MSFSX4MLGMZWOFMFUBRVRXHYV4S7GQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:48.969344229 (548h58m48.840427264s) [noise=true , address=77.90.31.19:4863 , piece_id=HJGJNIFUMGXDX2NS7VEB4NAHLTIPFKXVL6ZFGR6XWQCJCVXFD7JQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.306540067 (548h58m49.177623102s) [noise=true , address=145.79.227.44:4208 , piece_id=JUAG57OB2WR67TQMUE7WOWSB6XWUBLFTHCD7YSIXJJDEBOQTBYNQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.318819940 (548h58m49.189902975s) [noise=true , address=77.90.31.16:4757 , piece_id=U3RN7Q5FZIE6WP2QO4N5E5J3JVJUVYRZ25GIGAL4YLG2MJUUJYAA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.340178461 (548h58m49.211261496s) [noise=true , address=145.79.227.30:4787 , piece_id=WAHT65KVOTZ4J6YPYQUDDOHTOWDLLTQ6ONSIB4AZVEKQAO4NNC4A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363205077 (548h58m49.234288142s) [noise=true , address=145.79.227.43:4170 , piece_id=VCHJ3PGDNNKDILSCAKDTJLLVCYV446GS6JXWIWD3CZWBCSUR3M6A, seg_pos={0 0}, upload=481360]: Done uploading piece. err: <nil> Dec 28 12:27:49.363228532 (548h58m49.234311567s) [seg_pos={0 0}, upload=481360]: Segment reached optimal threshold of 14 pieces. Dec 28 12:27:49.363467547 (548h58m49.234550582s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12h5esfg6mw2c8y3AVpeFBkL5FfLAwYdjhcnQ9CJ8fjh61WLYjx): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363478808 (548h58m49.234561843s) [noise=true , address=45.91.116.4:4969 , piece_id=BYVHHRMS2GGHZ77X5J2Y4TYM5IWTCPKDP2GSCR7AV65M5H7EK5FQ, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363507082 (548h58m49.234590117s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363522251 (548h58m49.234605286s) [noise=true , address=77.90.31.10:4541 , piece_id=KU2RXVWG6K3NOKE2PBRA7A6OASJ4TWV6Q7L2LLDGE4A7I3QEF6NA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363643512 (548h58m49.234726577s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19TbrQjuN1EhBNVtha9EcKPSCAWyttfmHpec6QeF1S5zW6DRZs): protocol: expected piece hash; context canceled; EOF Dec 28 12:27:49.363652960 (548h58m49.234735995s) [noise=true , address=145.79.227.34:4924 , piece_id=T2GKDF5QR6ZFROVFAZKLKN6XX4FHVWLCRLSEMUKISNAVCHRPLSNA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363805260 (548h58m49.234888305s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->64.49.13.169:4706: use of closed network connection Dec 28 12:27:49.363812735 (548h58m49.23489577s) [noise=true , address=64.49.13.169:4706 , piece_id=D6RAJHZGLXCWY4IGUCXTUSLOSW6LL46UOYB2ONI5PIV5DMED63MA, seg_pos={0 0}, upload=481360]: Long tail cancellation due to optimal threshold Dec 28 12:27:49.363834976 (548h58m49.234918021s) [seg_pos={0 0}, upload=481360]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 12:27:49.363922574 (548h58m49.235005609s) [upload=481360]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:27:49.363928685 (548h58m49.23501172s) [upload=481360]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:27:49.653261754 (548h58m49.524344789s) [upload=481360]: Done uploading segments... Dec 28 12:27:49.653286421 (548h58m49.524369456s) [upload=481360]: Upload finished. err: <nil> Duration: 41.287088321s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X..XX.X....XX.X.....XX..............X.X.......X.X.......] duration: 5.911723272s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:49:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 5.911745054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X.XXX.XX.......X.XX.XX.X.XX.XX..XXX.X...X..XX.X...X....XXXXX] duration: 5.001087299s trace: https://jaeger.dev.storj.io/trace/ee8cedcd8db16ae FAIL (2025-12-28 12:49:50 UTC) orbiter warning: context deadline exceeded Duration: 5.001114132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X..XX.X....XX.X.....XX..............X.X.......X.X......] duration: 5.688263077s trace: https://jaeger.dev.storj.io/trace/5c4ccf8b511388d4 FAIL (2025-12-28 12:49:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, Piece ID: P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA) context deadline exceeded ======LOGS====== Dec 28 12:44:50.329765618 (549h15m50.200848664s) [upload=481592]: Starting upload Dec 28 12:44:50.329851052 (549h15m50.200934097s) [upload=481592]: Done starting upload Dec 28 12:44:50.330177152 (549h15m50.201260187s) [upload=481592]: Uploading segments... Dec 28 12:44:50.330239631 (549h15m50.201322677s) [upload=481592]: Got next segment. Inline: false Dec 28 12:44:50.330276822 (549h15m50.201359867s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 12:44:50.330282593 (549h15m50.201365638s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 12:44:50.577825859 (549h15m50.448908904s) [seg_pos={0 0}, upload=481592]: Begin upload segment... Dec 28 12:44:50.577891865 (549h15m50.4489749s) [seg_pos={0 0}, upload=481592]: Done begin upload segment. Dec 28 12:44:50.577929827 (549h15m50.449012872s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.577944595 (549h15m50.44902763s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578060365 (549h15m50.4491434s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578127362 (549h15m50.449210408s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.578661520 (549h15m50.449744575s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579027998 (549h15m50.450111043s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579396079 (549h15m50.450479114s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.579729243 (549h15m50.450812288s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.581914494 (549h15m50.452997529s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.582717934 (549h15m50.453800979s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584196971 (549h15m50.455280016s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.584889029 (549h15m50.455972094s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.585778311 (549h15m50.456861356s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.588126744 (549h15m50.459209779s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589533843 (549h15m50.460616888s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:50.589927482 (549h15m50.461010527s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Uploading piece... Dec 28 12:44:52.304551725 (549h15m52.17563476s) [upload=481592]: Next returned nil segment Dec 28 12:44:52.304576000 (549h15m52.175659066s) [upload=481592]: Waiting for error group managing segments... Dec 28 12:44:52.591624569 (549h15m52.462707615s) [noise=true , address=145.79.227.46:4601 , piece_id=JXMP4G2FAACJ7BDVWIWCRUMNPTHYQTWIWWB3YRW6WTEUCRXD3FBA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.596335470 (549h15m52.467418505s) [noise=true , address=145.79.227.42:4122 , piece_id=LHCQKTOK2J4XFQG3CGJVKXFVUV2CYBRSFB4V6YTDXOBBB4JXCHEQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.602935888 (549h15m52.474018933s) [noise=true , address=145.79.227.41:4087 , piece_id=7KWQY4DFIBWORXAGZHQCODW7PQ3VV6ZT5CXZDKOAYVFON7YSFU7A, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.608172720 (549h15m52.479255765s) [noise=true , address=145.79.227.20:4439 , piece_id=YJBY4ASMCPOMJFVHOFK4WLFFZQNHU3I3XYEDL2SRK5HUENPIQ4OA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609195428 (549h15m52.480278473s) [noise=true , address=145.79.227.4:4187 , piece_id=G677ZI6325UGLGNT4O6VBDAGPTHMGZNWYJD742PWYNNYJWQS4TVA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609600368 (549h15m52.480683403s) [noise=true , address=145.79.227.38:4977 , piece_id=NYSQET422WKEJRCZJNJJKPO4A5GKBSLUPLKQN2XPWSG4SUHQMTCQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.609757929 (549h15m52.480840964s) [noise=true , address=145.79.227.51:4409 , piece_id=52UOTCTB36A5ASXMR2UJTDFNUHKGVE5ZIMH5GJWPFXZQPWZI7SEA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.615184051 (549h15m52.486267086s) [noise=true , address=145.79.227.50:4374 , piece_id=2ZRBXIDZ57MSUYVLAQLSWWNENMVC3NZVLZGGGBPFY7P3MIUZXEXQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.616876273 (549h15m52.487959318s) [noise=true , address=145.79.227.45:4231 , piece_id=3IYV5KATGTR6QBZTHOVZFOHS42RKQIAYJP7W6PEFTKH2FRPLWUWA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.636903428 (549h15m52.507986473s) [noise=true , address=145.79.227.17:4971 , piece_id=R27FYXKOPICTQFGLH2X4O63WYDZGIZUS6GZ7PXDDOM5TVARO3YIQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.852247760 (549h15m52.723330805s) [noise=true , address=45.91.116.5:4298 , piece_id=P36NZBF57HP5VB3NH237YDGEB4VCGGBMUGMQFCBS5EVGKAXUTRZA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.859591744 (549h15m52.730674779s) [noise=true , address=45.91.116.9:4262 , piece_id=73XASGOXO7GW24QKHXBFDN6GGILVHVANRD2DADA52QFA4SILOQGQ, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943749244 (549h15m52.814832289s) [noise=true , address=45.91.116.13:4155 , piece_id=UHT6ISI6SXWCJNSMWD6HYUCDNLJ7ZE54SA5OTF3CNCBGCAYAJKGA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: <nil> Dec 28 12:44:52.943768901 (549h15m52.814851936s) [seg_pos={0 0}, upload=481592]: Segment reached optimal threshold of 13 pieces. Dec 28 12:44:52.943965816 (549h15m52.815048861s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.943973521 (549h15m52.815056566s) [noise=true , address=45.91.116.14:4190 , piece_id=5JLCEL2KMHHDK73DKF3IPX42EYDWFRD2WDR2UUS2Z2J3SN7GGT7Q, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944180515 (549h15m52.81526355s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 12:44:52.944199371 (549h15m52.815282406s) [noise=true , address=77.90.31.14:4664 , piece_id=JRHJS36CRDUJEWJ4UQGP2ZCZADBNYAOWI3RCSIXGEUB7G74G3DLA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944245208 (549h15m52.815328243s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->77.90.31.11:4556: use of closed network connection Dec 28 12:44:52.944258964 (549h15m52.815341999s) [noise=true , address=77.90.31.11:4556 , piece_id=DRTMV2GFGT45RKWHQSQMHXPVMERFDH6OBABI6DVJ6AVCHORUASMA, seg_pos={0 0}, upload=481592]: Long tail cancellation due to optimal threshold Dec 28 12:44:52.944282980 (549h15m52.815366015s) [seg_pos={0 0}, upload=481592]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 12:44:52.944414531 (549h15m52.815497566s) [upload=481592]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 12:44:52.944423027 (549h15m52.815506062s) [upload=481592]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 12:44:53.239033345 (549h15m53.11011638s) [upload=481592]: Done uploading segments... Dec 28 12:44:53.239054725 (549h15m53.11013776s) [upload=481592]: Upload finished. err: <nil> Duration: 5.688293306s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)