Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X.] duration: 6.124315731s trace: https://jaeger.dev.storj.io/trace/4d0231720e50e740 FAIL (2025-12-26 18:39:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:38:31.157821160 (507h9m31.028904205s) [upload=445034]: Starting upload Dec 26 18:38:31.157877146 (507h9m31.028960191s) [upload=445034]: Done starting upload Dec 26 18:38:31.157942992 (507h9m31.029026037s) [upload=445034]: Uploading segments... Dec 26 18:38:31.158059945 (507h9m31.02914298s) [upload=445034]: Got next segment. Inline: false Dec 26 18:38:31.158089972 (507h9m31.029173018s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:38:31.158094451 (507h9m31.029177486s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:38:31.404855186 (507h9m31.275938231s) [seg_pos={0 0}, upload=445034]: Begin upload segment... Dec 26 18:38:31.404909629 (507h9m31.275992664s) [seg_pos={0 0}, upload=445034]: Done begin upload segment. Dec 26 18:38:31.404939226 (507h9m31.276022261s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.405917509 (507h9m31.277000554s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.406590080 (507h9m31.277673115s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407173159 (507h9m31.278256195s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407546741 (507h9m31.278629786s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407938116 (507h9m31.279021151s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.408758859 (507h9m31.279841894s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409240627 (507h9m31.280323662s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409752571 (507h9m31.280835626s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410632658 (507h9m31.281715693s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410855772 (507h9m31.281938817s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.411905451 (507h9m31.282988516s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.412072970 (507h9m31.283156015s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.413211528 (507h9m31.284294563s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.414546780 (507h9m31.285629815s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.415591630 (507h9m31.286674665s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:32.593710866 (507h9m32.464793931s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 18:38:33.121407613 (507h9m32.992490648s) [upload=445034]: Next returned nil segment Dec 26 18:38:33.121426890 (507h9m32.992509925s) [upload=445034]: Waiting for error group managing segments... Dec 26 18:38:33.413463338 (507h9m33.284546373s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.414417395 (507h9m33.28550043s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.423567389 (507h9m33.294650424s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.432953442 (507h9m33.304036477s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436006996 (507h9m33.307090031s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436422918 (507h9m33.307505953s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436849921 (507h9m33.307932966s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.449704228 (507h9m33.320787263s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.460416667 (507h9m33.331499712s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.517087077 (507h9m33.388170112s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.600393006 (507h9m33.471476051s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.662483417 (507h9m33.533566472s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783839309 (507h9m33.654922344s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783857174 (507h9m33.654940209s) [seg_pos={0 0}, upload=445034]: Segment reached optimal threshold of 13 pieces. Dec 26 18:38:33.784044630 (507h9m33.655127665s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 18:38:33.784050812 (507h9m33.655133847s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784255181 (507h9m33.655338216s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection Dec 26 18:38:33.784269758 (507h9m33.655352794s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784299335 (507h9m33.65538237s) [seg_pos={0 0}, upload=445034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:38:33.784380079 (507h9m33.655463124s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:38:33.784384908 (507h9m33.655467953s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:38:34.099088118 (507h9m33.970171153s) [upload=445034]: Done uploading segments... Dec 26 18:38:34.099106454 (507h9m33.970189499s) [upload=445034]: Upload finished. err: <nil> Duration: 6.124332171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X..X.....X...XX.........X.....X..X..X.X.X.....XXX.X] duration: 30.002530261s trace: https://jaeger.dev.storj.io/trace/4d0231720e50e740 FAIL (2025-12-26 18:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:38:31.157821160 (507h9m31.028904205s) [upload=445034]: Starting upload Dec 26 18:38:31.157877146 (507h9m31.028960191s) [upload=445034]: Done starting upload Dec 26 18:38:31.157942992 (507h9m31.029026037s) [upload=445034]: Uploading segments... Dec 26 18:38:31.158059945 (507h9m31.02914298s) [upload=445034]: Got next segment. Inline: false Dec 26 18:38:31.158089972 (507h9m31.029173018s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:38:31.158094451 (507h9m31.029177486s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:38:31.404855186 (507h9m31.275938231s) [seg_pos={0 0}, upload=445034]: Begin upload segment... Dec 26 18:38:31.404909629 (507h9m31.275992664s) [seg_pos={0 0}, upload=445034]: Done begin upload segment. Dec 26 18:38:31.404939226 (507h9m31.276022261s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.405917509 (507h9m31.277000554s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.406590080 (507h9m31.277673115s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407173159 (507h9m31.278256195s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407546741 (507h9m31.278629786s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.407938116 (507h9m31.279021151s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.408758859 (507h9m31.279841894s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409240627 (507h9m31.280323662s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.409752571 (507h9m31.280835626s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410632658 (507h9m31.281715693s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.410855772 (507h9m31.281938817s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.411905451 (507h9m31.282988516s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.412072970 (507h9m31.283156015s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.413211528 (507h9m31.284294563s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.414546780 (507h9m31.285629815s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:31.415591630 (507h9m31.286674665s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Uploading piece... Dec 26 18:38:32.593710866 (507h9m32.464793931s) [noise=true , address=77.90.31.12:4589 , piece_id=4IGKBXF3SOCISAXEAKCSWWZ66BLIQ7NQ5L4YYMITLMCFFDSK3DXA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 18:38:33.121407613 (507h9m32.992490648s) [upload=445034]: Next returned nil segment Dec 26 18:38:33.121426890 (507h9m32.992509925s) [upload=445034]: Waiting for error group managing segments... Dec 26 18:38:33.413463338 (507h9m33.284546373s) [noise=true , address=145.79.227.45:4229 , piece_id=YWGQ3PU6GNCTDD3ECFONV2GROLUZNRPCGHJN7UP7ZDJITO5N56FA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.414417395 (507h9m33.28550043s) [noise=true , address=145.79.227.46:4601 , piece_id=JS6TZ4ILXJ6ERH3HWMTSDNWRSPCTP25ND7UIFO7B7CDHHHAQKE4Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.423567389 (507h9m33.294650424s) [noise=true , address=145.79.227.32:4836 , piece_id=2R6WOS7NHLSL3IECRA4DTPQCA2AIGJE4VZWR472QHUXBYGWED45Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.432953442 (507h9m33.304036477s) [noise=true , address=145.79.227.52:4445 , piece_id=ZPIMEROEBTNLTCAQF42GIRND2LZPUFECG5QP2GPJUUHBG47BFTOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436006996 (507h9m33.307090031s) [noise=true , address=145.79.227.51:4409 , piece_id=2I5ZV3TP3NXSZS7KVH4JBSO5VBWQGKRPDS3MFRR2YLH3TLM33SSQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436422918 (507h9m33.307505953s) [noise=true , address=145.79.227.36:4519 , piece_id=CYT4FFVVOTUIKTHMRM5VIJ4O5QR6AAB3MMZGQJ2DHLHV7BSHXC3Q, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.436849921 (507h9m33.307932966s) [noise=true , address=145.79.227.42:4121 , piece_id=4U7MBYKX2BMTHU3FPXNWQONZLPAH5I5REX4MD2IGPEK6DXG6COAA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.449704228 (507h9m33.320787263s) [noise=true , address=145.79.227.29:4943 , piece_id=L5TE5QVCMG6IZ5FE7HOXRBWJMUAI3QA6D6JOCH52AEGIX7JU3XDQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.460416667 (507h9m33.331499712s) [noise=true , address=145.79.227.16:4909 , piece_id=4CQ3ZCDAGEWPQOM4VANX45TTWFOGKJRXWVAUNUR6FD4EZHZMK2SQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.517087077 (507h9m33.388170112s) [noise=true , address=145.79.227.2:4114 , piece_id=36QOULUKE6SBVZIKHJBIPB36OBVOFXAJZ6DVNJHTQVM4FIEMNFOA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.600393006 (507h9m33.471476051s) [noise=true , address=77.90.31.13:4627 , piece_id=46RM6UI6UNSEYAYTYRE5RTI6TTJSMKZQVI2OOK2FVYBWM5T6X4JA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.662483417 (507h9m33.533566472s) [noise=true , address=45.91.116.14:4190 , piece_id=RSD64GG2KX27FKVD5XXRA4I3TFEA76ZNBY5HYTPKPLYLPCKGVUKA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783839309 (507h9m33.654922344s) [noise=true , address=45.91.116.8:4048 , piece_id=DUK7YWADLEYMV2YVL3R6FE6MKVNI3EFECHIXONPRU3K2JIZBP3VA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: <nil> Dec 26 18:38:33.783857174 (507h9m33.654940209s) [seg_pos={0 0}, upload=445034]: Segment reached optimal threshold of 13 pieces. Dec 26 18:38:33.784044630 (507h9m33.655127665s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 26 18:38:33.784050812 (507h9m33.655133847s) [noise=true , address=45.91.116.11:4120 , piece_id=PYWGMEJ3SJ56CE4WW6N22XQUK5ICC2LJC5ICEHDNA7RP44SU5ZJA, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784255181 (507h9m33.655338216s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51964->45.91.116.12:4227: use of closed network connection Dec 26 18:38:33.784269758 (507h9m33.655352794s) [noise=true , address=45.91.116.12:4227 , piece_id=X6LTBYJNVPHRDRKFTYTDDY6N33YWIX3JCME6JLIDFKI52YKSRPLQ, seg_pos={0 0}, upload=445034]: Long tail cancellation due to optimal threshold Dec 26 18:38:33.784299335 (507h9m33.65538237s) [seg_pos={0 0}, upload=445034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:38:33.784380079 (507h9m33.655463124s) [upload=445034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:38:33.784384908 (507h9m33.655467953s) [upload=445034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:38:34.099088118 (507h9m33.970171153s) [upload=445034]: Done uploading segments... Dec 26 18:38:34.099106454 (507h9m33.970189499s) [upload=445034]: Upload finished. err: <nil> Duration: 30.002552534s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.........X..X.....X...XX.........X.....X..X..X.X.X.....XXX.] duration: 7.210502173s trace: https://jaeger.dev.storj.io/trace/3b45fd7658be0262 FAIL (2025-12-26 18:38:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q) context deadline exceeded ======LOGS====== Dec 26 18:37:19.121626906 (507h8m18.992709951s) [upload=445016]: Starting upload Dec 26 18:37:19.121685578 (507h8m18.992768623s) [upload=445016]: Done starting upload Dec 26 18:37:19.121785598 (507h8m18.992868634s) [upload=445016]: Uploading segments... Dec 26 18:37:19.121920165 (507h8m18.99300321s) [upload=445016]: Got next segment. Inline: false Dec 26 18:37:19.121963387 (507h8m18.993046422s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:37:19.121968106 (507h8m18.993051141s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:37:19.372302056 (507h8m19.243385101s) [seg_pos={0 0}, upload=445016]: Begin upload segment... Dec 26 18:37:19.372356729 (507h8m19.243439764s) [seg_pos={0 0}, upload=445016]: Done begin upload segment. Dec 26 18:37:19.372392647 (507h8m19.243475682s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.372401365 (507h8m19.24348441s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373123118 (507h8m19.244206163s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373405486 (507h8m19.244488521s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373656364 (507h8m19.2447394s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373679438 (507h8m19.244762493s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373951776 (507h8m19.245034812s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374182646 (507h8m19.245265682s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374233853 (507h8m19.245316889s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374509018 (507h8m19.245592053s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374862831 (507h8m19.245945866s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375174074 (507h8m19.246257119s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375501438 (507h8m19.246584483s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375841154 (507h8m19.246924199s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376217330 (507h8m19.247300375s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376820149 (507h8m19.247903254s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:21.089122453 (507h8m20.960205498s) [upload=445016]: Next returned nil segment Dec 26 18:37:21.089141359 (507h8m20.960224394s) [upload=445016]: Waiting for error group managing segments... Dec 26 18:37:21.363581731 (507h8m21.234664766s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.368429613 (507h8m21.239512658s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.370357303 (507h8m21.241440348s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.379265586 (507h8m21.250348621s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.394533419 (507h8m21.265616464s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.402674891 (507h8m21.273757966s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.403498369 (507h8m21.274581414s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.435802712 (507h8m21.306885747s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.436628545 (507h8m21.30771158s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.649643122 (507h8m21.520726167s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.739242897 (507h8m21.610325932s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.979515096 (507h8m21.850598141s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352800938 (507h8m22.223884013s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352832238 (507h8m22.223915283s) [seg_pos={0 0}, upload=445016]: Segment reached optimal threshold of 13 pieces. Dec 26 18:37:22.353106590 (507h8m22.224189645s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection Dec 26 18:37:22.353116970 (507h8m22.224200025s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353157867 (507h8m22.224240912s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353168328 (507h8m22.224251373s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353342600 (507h8m22.224425655s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353353791 (507h8m22.224436866s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353381794 (507h8m22.224464839s) [seg_pos={0 0}, upload=445016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:37:22.353476835 (507h8m22.224559891s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:37:22.353484730 (507h8m22.224567785s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:37:22.655223604 (507h8m22.526306649s) [upload=445016]: Done uploading segments... Dec 26 18:37:22.655247399 (507h8m22.526330434s) [upload=445016]: Upload finished. err: <nil> Duration: 7.210585762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.........X..X.....X...XX.........X.....X..X..X.X.X.....XXX] duration: 30.003966518s trace: https://jaeger.dev.storj.io/trace/3b45fd7658be0262 FAIL (2025-12-26 18:37:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA) context deadline exceeded; piecestore: (Node ID: 1kgUqvw1r3gSKw1bNjzys3AiAJXdkAnSafmvcPgNVPBbN9JS1d, Piece ID: NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q) context deadline exceeded ======LOGS====== Dec 26 18:37:19.121626906 (507h8m18.992709951s) [upload=445016]: Starting upload Dec 26 18:37:19.121685578 (507h8m18.992768623s) [upload=445016]: Done starting upload Dec 26 18:37:19.121785598 (507h8m18.992868634s) [upload=445016]: Uploading segments... Dec 26 18:37:19.121920165 (507h8m18.99300321s) [upload=445016]: Got next segment. Inline: false Dec 26 18:37:19.121963387 (507h8m18.993046422s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:37:19.121968106 (507h8m18.993051141s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:37:19.372302056 (507h8m19.243385101s) [seg_pos={0 0}, upload=445016]: Begin upload segment... Dec 26 18:37:19.372356729 (507h8m19.243439764s) [seg_pos={0 0}, upload=445016]: Done begin upload segment. Dec 26 18:37:19.372392647 (507h8m19.243475682s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.372401365 (507h8m19.24348441s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373123118 (507h8m19.244206163s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373405486 (507h8m19.244488521s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373656364 (507h8m19.2447394s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373679438 (507h8m19.244762493s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.373951776 (507h8m19.245034812s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374182646 (507h8m19.245265682s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374233853 (507h8m19.245316889s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374509018 (507h8m19.245592053s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.374862831 (507h8m19.245945866s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375174074 (507h8m19.246257119s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375501438 (507h8m19.246584483s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.375841154 (507h8m19.246924199s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376217330 (507h8m19.247300375s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:19.376820149 (507h8m19.247903254s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Uploading piece... Dec 26 18:37:21.089122453 (507h8m20.960205498s) [upload=445016]: Next returned nil segment Dec 26 18:37:21.089141359 (507h8m20.960224394s) [upload=445016]: Waiting for error group managing segments... Dec 26 18:37:21.363581731 (507h8m21.234664766s) [noise=true , address=145.79.227.43:4158 , piece_id=4Z5DAXGBYWCZQH4PLYZDJL4OOMA7TTXAUXZ3HTSORMDAB7HHE3FQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.368429613 (507h8m21.239512658s) [noise=true , address=145.79.227.21:4475 , piece_id=IRI7GQFKTWPI2NRYCEM2PVDN46I6IFN4ONQOJV3SSMLUJXGCD7GQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.370357303 (507h8m21.241440348s) [noise=true , address=145.79.227.37:4565 , piece_id=Y22BCACQP6GKMNCFN7L2AWGJHXEBEQYGCW7P6UHQTN5S4SAMH4JQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.379265586 (507h8m21.250348621s) [noise=true , address=145.79.227.48:4303 , piece_id=ANWVQPQ4HARTRLRYO3X2FGXWBTN35YCMRQZXSK534FZOGNAZLNPQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.394533419 (507h8m21.265616464s) [noise=true , address=145.79.227.8:4330 , piece_id=PSXZGJ3UMIYBVRDKCXRPR63ZYBUKZ75C6O24CAHSQYZK7KO2LPXQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.402674891 (507h8m21.273757966s) [noise=true , address=145.79.227.51:4409 , piece_id=AKO6JWAXFOMWPQHM7SOVGX4HABVCBHYTSKCUGWNP6N2VLEBDIRJQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.403498369 (507h8m21.274581414s) [noise=true , address=145.79.227.46:4602 , piece_id=NK6JINBNQOSTK6W4CDVR5SEUJVCQOLJE2KYDLAHID2G63YQTQCMQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.435802712 (507h8m21.306885747s) [noise=true , address=145.79.227.2:4115 , piece_id=EU56CW4GG76TO67THVLBF6RZAT6D4GD6MPYSRDIDY44IBIHBBXNQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.436628545 (507h8m21.30771158s) [noise=true , address=145.79.227.12:4767 , piece_id=T7A4IXELU3AYIXDV2W2EHFAFLN6XQC7UIT76F4OH3WYDPJ2RNSYQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.649643122 (507h8m21.520726167s) [noise=true , address=145.79.227.41:4086 , piece_id=3IUK67FIMMKTR7NXHRSHWM7JCOXZFICZT6PHWXDKBUAWSJC23VMA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.739242897 (507h8m21.610325932s) [noise=true , address=45.91.116.12:4227 , piece_id=PWCCOYLWEYFEXKCWUOKVRUN6HTENOWNKAV75WY2L4AY34N66V35Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:21.979515096 (507h8m21.850598141s) [noise=true , address=77.90.31.10:4518 , piece_id=XN6HWG2K6V4DMLR42H2BOPVC2BE6RGNBTNGUQNOC6FYAYD7LPQ2Q, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352800938 (507h8m22.223884013s) [noise=true , address=45.91.116.6:4976 , piece_id=NXLUQNJVFLGBOEPPCO2ZZOAECZAZT26SFU3KBFVBYUNZYLCAKRWA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: <nil> Dec 26 18:37:22.352832238 (507h8m22.223915283s) [seg_pos={0 0}, upload=445016]: Segment reached optimal threshold of 13 pieces. Dec 26 18:37:22.353106590 (507h8m22.224189645s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41048->145.79.227.14:4839: use of closed network connection Dec 26 18:37:22.353116970 (507h8m22.224200025s) [noise=true , address=145.79.227.14:4839 , piece_id=MAA4EEX7BSJKPCDQOGUO2QRT5O6ILJ3KFK7NOFSHWG357VAY737A, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353157867 (507h8m22.224240912s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353168328 (507h8m22.224251373s) [noise=true , address=45.91.116.5:4300 , piece_id=JMCPZRHD52SRLTLYJU3ACH37IBFMMHWEABV265SE5STCCOPHN4UA, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353342600 (507h8m22.224425655s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:37:22.353353791 (507h8m22.224436866s) [noise=true , address=77.90.31.16:4736 , piece_id=KO2NNFY4YDMRWD5BYBKHF7UUNAE2G5KICGDWDIHEJNFGUF7NIJBQ, seg_pos={0 0}, upload=445016]: Long tail cancellation due to optimal threshold Dec 26 18:37:22.353381794 (507h8m22.224464839s) [seg_pos={0 0}, upload=445016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:37:22.353476835 (507h8m22.224559891s) [upload=445016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:37:22.353484730 (507h8m22.224567785s) [upload=445016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:37:22.655223604 (507h8m22.526306649s) [upload=445016]: Done uploading segments... Dec 26 18:37:22.655247399 (507h8m22.526330434s) [upload=445016]: Upload finished. err: <nil> Duration: 30.00398931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX.........X..X.....X...XX.........X.....X..X..X.X.X.....XX] duration: 30.003867418s trace: https://jaeger.dev.storj.io/trace/60e16220d283e446 FAIL (2025-12-26 18:36:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: Q767FIFGU7H2Y4QFF2R55DH7FVZWG5MR5DVLZZVKLZWONVOCMOEQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: WFPLES7OZICSROZHQLMTFUZNGHK5JTVRM3GOAQB77H5V6TUKPAUQ) context deadline exceeded ======LOGS====== Dec 26 18:36:18.561272816 (507h7m18.432355861s) [upload=445001]: Starting upload Dec 26 18:36:18.561380051 (507h7m18.432463107s) [upload=445001]: Done starting upload Dec 26 18:36:18.561425408 (507h7m18.432508453s) [upload=445001]: Uploading segments... Dec 26 18:36:18.561658351 (507h7m18.432741396s) [upload=445001]: Got next segment. Inline: false Dec 26 18:36:18.561708096 (507h7m18.432791141s) [upload=445001]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:36:18.561714317 (507h7m18.432797362s) [upload=445001]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:36:18.815151536 (507h7m18.686234571s) [seg_pos={0 0}, upload=445001]: Begin upload segment... Dec 26 18:36:18.815218373 (507h7m18.686301419s) [seg_pos={0 0}, upload=445001]: Done begin upload segment. Dec 26 18:36:18.815399159 (507h7m18.686482204s) [noise=true , address=145.79.227.2:4115 , piece_id=2ORBM67MVBOGC5RSRX4ESFWV4A44L3D4ZOC3G54CGKRPAWRDVITQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.815463340 (507h7m18.686546375s) [noise=true , address=77.90.31.19:4844 , piece_id=Q767FIFGU7H2Y4QFF2R55DH7FVZWG5MR5DVLZZVKLZWONVOCMOEQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.816148455 (507h7m18.6872315s) [noise=true , address=145.79.227.36:4517 , piece_id=FDASGOQS6SQ6XMEBMVXUTEVDIXIFECJEVWROWI6Q3DGGNPWDSJZQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.816925925 (507h7m18.68800897s) [noise=true , address=145.79.227.8:4330 , piece_id=ZJQL3S44BP7J3Z5MTG6RM4727DIGBDPSFVEW7XSC3U3SHGXVQNJA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.817498035 (507h7m18.68858107s) [noise=true , address=45.91.116.5:4297 , piece_id=VPHNE435FLAOQNQJAL76RZW5UBM7LHIFGPA34QZVWF467VJNYUDA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.818185514 (507h7m18.689268549s) [noise=true , address=145.79.227.11:4729 , piece_id=S7EBFZ6JYWSJZSY24PQ5WKYV7W6RDPJV7SVM3WC6UZ4EXU2QE6AQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.818761961 (507h7m18.689845006s) [noise=true , address=45.91.116.13:4153 , piece_id=EQ4AHINMJQ7W2QTMF4BLT2TG7PCKBKXOVHOZVILDSOUYGW4INMDQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.819104263 (507h7m18.690187308s) [noise=true , address=45.91.116.14:4190 , piece_id=BDGLJ3SAP7ERSBFBQAOMUHD2IVPYUFMC6FPE6KDGVPM6HBGI2GKQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.819529813 (507h7m18.690612848s) [noise=true , address=45.91.116.12:4225 , piece_id=Z74ZZ5HSG7AU754ZZFHO4O4FBTW52MQZM3F6PCFB3XIVANL6S5OQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.819797193 (507h7m18.690880248s) [noise=true , address=145.79.227.47:4265 , piece_id=EVZ4VW2SDCYT6HRUSPLGWLSRUHXB4TUJJGPQCRQFSDW3LDJFWZ5Q, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.820127171 (507h7m18.691210216s) [noise=true , address=45.91.116.10:4084 , piece_id=WFPLES7OZICSROZHQLMTFUZNGHK5JTVRM3GOAQB77H5V6TUKPAUQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.820500091 (507h7m18.691583136s) [noise=true , address=145.79.227.4:4185 , piece_id=SHMWXWXU7FYXWBQAPTDLRRW3K42LJUZF5DOKSF2NFRHPYZ5SOEIA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.820709660 (507h7m18.691792695s) [noise=true , address=45.91.116.4:4937 , piece_id=S26AQ32EUAN3J7WX7IBIHGPOW36XQ7Z3ZYYZZOA42KXNAU5U4WMQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.821017587 (507h7m18.692100632s) [noise=true , address=145.79.227.42:4122 , piece_id=VJWHP3KIMEOAGHNDYOGPHXSLYBNYVYDGJ5NYLK5C2JOLOVTATZPA, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.821316797 (507h7m18.692399832s) [noise=true , address=45.91.116.6:4975 , piece_id=K3FG3BBKEMLUGKBOGZVK4BMOM5IIQ7XYGO47OSN2W664PKZXPLKQ, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:18.821783866 (507h7m18.692866911s) [noise=true , address=145.79.227.48:4301 , piece_id=23TYBE2NUXCLCMGWFGCBWMEIJELOOUH22PFJ6ZNICINGDZINTR3A, seg_pos={0 0}, upload=445001]: Uploading piece... Dec 26 18:36:20.527472836 (507h7m20.398555871s) [upload=445001]: Next returned nil segment Dec 26 18:36:20.527485421 (507h7m20.398568456s) [upload=445001]: Waiting for error group managing segments... Dec 26 18:36:20.820006592 (507h7m20.691089637s) [noise=true , address=145.79.227.11:4729 , piece_id=S7EBFZ6JYWSJZSY24PQ5WKYV7W6RDPJV7SVM3WC6UZ4EXU2QE6AQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.824369199 (507h7m20.695452234s) [noise=true , address=145.79.227.36:4517 , piece_id=FDASGOQS6SQ6XMEBMVXUTEVDIXIFECJEVWROWI6Q3DGGNPWDSJZQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.827186353 (507h7m20.698269399s) [noise=true , address=145.79.227.48:4301 , piece_id=23TYBE2NUXCLCMGWFGCBWMEIJELOOUH22PFJ6ZNICINGDZINTR3A, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.830331523 (507h7m20.701414558s) [noise=true , address=145.79.227.8:4330 , piece_id=ZJQL3S44BP7J3Z5MTG6RM4727DIGBDPSFVEW7XSC3U3SHGXVQNJA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.833317318 (507h7m20.704400363s) [noise=true , address=145.79.227.42:4122 , piece_id=VJWHP3KIMEOAGHNDYOGPHXSLYBNYVYDGJ5NYLK5C2JOLOVTATZPA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:20.834319486 (507h7m20.705402531s) [noise=true , address=145.79.227.47:4265 , piece_id=EVZ4VW2SDCYT6HRUSPLGWLSRUHXB4TUJJGPQCRQFSDW3LDJFWZ5Q, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.051595078 (507h7m20.922678123s) [noise=true , address=45.91.116.4:4937 , piece_id=S26AQ32EUAN3J7WX7IBIHGPOW36XQ7Z3ZYYZZOA42KXNAU5U4WMQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.111565442 (507h7m20.982648477s) [noise=true , address=45.91.116.5:4297 , piece_id=VPHNE435FLAOQNQJAL76RZW5UBM7LHIFGPA34QZVWF467VJNYUDA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.136100971 (507h7m21.007184016s) [noise=true , address=45.91.116.13:4153 , piece_id=EQ4AHINMJQ7W2QTMF4BLT2TG7PCKBKXOVHOZVILDSOUYGW4INMDQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.137392211 (507h7m21.008475256s) [noise=true , address=77.90.31.19:4844 , piece_id=Q767FIFGU7H2Y4QFF2R55DH7FVZWG5MR5DVLZZVKLZWONVOCMOEQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.151964037 (507h7m21.023047073s) [noise=true , address=45.91.116.10:4084 , piece_id=WFPLES7OZICSROZHQLMTFUZNGHK5JTVRM3GOAQB77H5V6TUKPAUQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.206211344 (507h7m21.077294389s) [noise=true , address=45.91.116.14:4190 , piece_id=BDGLJ3SAP7ERSBFBQAOMUHD2IVPYUFMC6FPE6KDGVPM6HBGI2GKQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.240102849 (507h7m21.111185894s) [noise=true , address=45.91.116.6:4975 , piece_id=K3FG3BBKEMLUGKBOGZVK4BMOM5IIQ7XYGO47OSN2W664PKZXPLKQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: <nil> Dec 26 18:36:21.240129961 (507h7m21.111212996s) [seg_pos={0 0}, upload=445001]: Segment reached optimal threshold of 13 pieces. Dec 26 18:36:21.240350941 (507h7m21.111433976s) [noise=true , address=145.79.227.4:4185 , piece_id=SHMWXWXU7FYXWBQAPTDLRRW3K42LJUZF5DOKSF2NFRHPYZ5SOEIA, seg_pos={0 0}, upload=445001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: expected piece hash; context canceled; EOF Dec 26 18:36:21.240360350 (507h7m21.111443385s) [noise=true , address=145.79.227.4:4185 , piece_id=SHMWXWXU7FYXWBQAPTDLRRW3K42LJUZF5DOKSF2NFRHPYZ5SOEIA, seg_pos={0 0}, upload=445001]: Long tail cancellation due to optimal threshold Dec 26 18:36:21.240562724 (507h7m21.11164577s) [noise=true , address=45.91.116.12:4225 , piece_id=Z74ZZ5HSG7AU754ZZFHO4O4FBTW52MQZM3F6PCFB3XIVANL6S5OQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 26 18:36:21.240582062 (507h7m21.111665097s) [noise=true , address=45.91.116.12:4225 , piece_id=Z74ZZ5HSG7AU754ZZFHO4O4FBTW52MQZM3F6PCFB3XIVANL6S5OQ, seg_pos={0 0}, upload=445001]: Long tail cancellation due to optimal threshold Dec 26 18:36:21.240618391 (507h7m21.111701426s) [noise=true , address=145.79.227.2:4115 , piece_id=2ORBM67MVBOGC5RSRX4ESFWV4A44L3D4ZOC3G54CGKRPAWRDVITQ, seg_pos={0 0}, upload=445001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:39560->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:39560->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39560->145.79.227.2:4115: use of closed network connection Dec 26 18:36:21.240625835 (507h7m21.11170889s) [noise=true , address=145.79.227.2:4115 , piece_id=2ORBM67MVBOGC5RSRX4ESFWV4A44L3D4ZOC3G54CGKRPAWRDVITQ, seg_pos={0 0}, upload=445001]: Long tail cancellation due to optimal threshold Dec 26 18:36:21.240643208 (507h7m21.111726254s) [seg_pos={0 0}, upload=445001]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:36:21.240716929 (507h7m21.111799974s) [upload=445001]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:36:21.240721447 (507h7m21.111804482s) [upload=445001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:36:21.538036307 (507h7m21.409119352s) [upload=445001]: Done uploading segments... Dec 26 18:36:21.538059622 (507h7m21.409142657s) [upload=445001]: Upload finished. err: <nil> Duration: 30.003888998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX.........X..X.....X...XX.........X.....X..X..X.X.X.....X] duration: 30.002338367s trace: https://jaeger.dev.storj.io/trace/691da54eaa5c6952 FAIL (2025-12-26 18:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:35:08.500462979 (507h6m8.371546024s) [upload=444981]: Starting upload Dec 26 18:35:08.500546970 (507h6m8.371630025s) [upload=444981]: Done starting upload Dec 26 18:35:08.500640638 (507h6m8.371723673s) [upload=444981]: Uploading segments... Dec 26 18:35:08.500788039 (507h6m8.371871074s) [upload=444981]: Got next segment. Inline: false Dec 26 18:35:08.500834036 (507h6m8.371917081s) [upload=444981]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:35:08.500840549 (507h6m8.371923584s) [upload=444981]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:35:08.751122910 (507h6m8.622205955s) [seg_pos={0 0}, upload=444981]: Begin upload segment... Dec 26 18:35:08.751289948 (507h6m8.622372983s) [seg_pos={0 0}, upload=444981]: Done begin upload segment. Dec 26 18:35:08.751332198 (507h6m8.622415233s) [noise=true , address=145.79.227.8:4330 , piece_id=KWHBAURRCQEXOCB4XADLOOUDZIKQ7KYVAKEHDBRG4V2YUGTM7VFQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.755222266 (507h6m8.626305321s) [noise=true , address=145.79.227.21:4474 , piece_id=LUUPX2QTVJNDJE7GU6TTWE5KI7KXVQODJN6245XXI2CDZOMOZQFA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.759907798 (507h6m8.630990843s) [noise=true , address=145.79.227.38:4979 , piece_id=TTBTN7CTRJU26U3B6AQKYE2KXCUXXY45AHOMBPJIOM2OGA6EZLJA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.760161210 (507h6m8.631244255s) [noise=true , address=77.90.31.13:4627 , piece_id=FAFKNLLQWAPM6EIU7LGCHWFP4I6MSJQJEVCQQSZJZFNGQND2YQ5Q, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.762002797 (507h6m8.633085852s) [noise=true , address=145.79.227.17:4972 , piece_id=USNUVEGUL246EVFO66TLRDVNIMH6OBN2MTPE5M7H73ERX2JDSHBA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.762182179 (507h6m8.633265214s) [noise=true , address=145.79.227.33:4870 , piece_id=NGODTYLGR2ZYWAZNRDBG5IQ5UAZVNJ2ETOEHTMSEEW2FHB5UI7BA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.763686132 (507h6m8.634769177s) [noise=true , address=145.79.227.48:4301 , piece_id=ARQNMS6TPFPZRJJXGA3XLTG4DV5VQEY67EVSQPLHRRIC2KHO5CSA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.764391937 (507h6m8.635474982s) [noise=true , address=45.91.116.10:4084 , piece_id=YYBEWDUMQDXAL6K5NX3G7ZHATAXMDZJXK3SLQ456OPU7MSNV6WPQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.765191059 (507h6m8.636274104s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5227ZHY2JHTRO6QS3SOY7SQEBWCEH7RVLRY6LLYT5WDYBY7P2PA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.766238232 (507h6m8.637321278s) [noise=true , address=145.79.227.52:4445 , piece_id=LLDIZQQDYSWK55YABI2EVGESCTBFN76B7MF6ZV5B6WSZ7HZKMFWA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.766917786 (507h6m8.638000831s) [noise=true , address=45.91.116.6:4973 , piece_id=KG24MG5HRPDIRSHJVCOSPFTLAW33FSZGDT723IOWBGYZ4XLWWCBQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.768061193 (507h6m8.639144238s) [noise=true , address=145.79.227.35:4483 , piece_id=6KKDTL7KU2QDX2IX5T6KMFWC7Z4G2EVK5R62BPEN3UFALTH2YYSQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.768335235 (507h6m8.63941828s) [noise=true , address=145.79.227.37:4565 , piece_id=KUZANZAJOAWF2WR5B4DASW7XJHRSWY3GBBJ2UKLE46J2ULROD2AQ, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.769230450 (507h6m8.640313485s) [noise=true , address=77.90.31.16:4735 , piece_id=G5DCHTYQ4YDVOIY3UHCEWMOPXOHZHCR3LCVAMJBTEYKPS5MAQLZA, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.773034584 (507h6m8.644117619s) [noise=true , address=77.90.31.14:4663 , piece_id=KYWCYO5ZLTD3VZDLX4WGUPOIY7HLSYWL3XHRT7L4DWA2TMFVKJ4A, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:08.774840763 (507h6m8.645923798s) [noise=true , address=145.79.227.36:4519 , piece_id=YH3E6JN7HFLAXE7QFOWSEJIF42HGNAAJVHNR46LVTK5MOQM3YG5A, seg_pos={0 0}, upload=444981]: Uploading piece... Dec 26 18:35:10.475630375 (507h6m10.34671342s) [upload=444981]: Next returned nil segment Dec 26 18:35:10.475649411 (507h6m10.346732456s) [upload=444981]: Waiting for error group managing segments... Dec 26 18:35:10.774340622 (507h6m10.645423667s) [noise=true , address=145.79.227.48:4301 , piece_id=ARQNMS6TPFPZRJJXGA3XLTG4DV5VQEY67EVSQPLHRRIC2KHO5CSA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.779162854 (507h6m10.650245909s) [noise=true , address=145.79.227.8:4330 , piece_id=KWHBAURRCQEXOCB4XADLOOUDZIKQ7KYVAKEHDBRG4V2YUGTM7VFQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.783109640 (507h6m10.654192685s) [noise=true , address=145.79.227.36:4519 , piece_id=YH3E6JN7HFLAXE7QFOWSEJIF42HGNAAJVHNR46LVTK5MOQM3YG5A, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.787047348 (507h6m10.658130383s) [noise=true , address=145.79.227.37:4565 , piece_id=KUZANZAJOAWF2WR5B4DASW7XJHRSWY3GBBJ2UKLE46J2ULROD2AQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.789705799 (507h6m10.660788834s) [noise=true , address=145.79.227.21:4474 , piece_id=LUUPX2QTVJNDJE7GU6TTWE5KI7KXVQODJN6245XXI2CDZOMOZQFA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.799441458 (507h6m10.670524493s) [noise=true , address=145.79.227.33:4870 , piece_id=NGODTYLGR2ZYWAZNRDBG5IQ5UAZVNJ2ETOEHTMSEEW2FHB5UI7BA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.800499753 (507h6m10.671582788s) [noise=true , address=145.79.227.52:4445 , piece_id=LLDIZQQDYSWK55YABI2EVGESCTBFN76B7MF6ZV5B6WSZ7HZKMFWA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.809186094 (507h6m10.680269129s) [noise=true , address=145.79.227.17:4972 , piece_id=USNUVEGUL246EVFO66TLRDVNIMH6OBN2MTPE5M7H73ERX2JDSHBA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:10.850078059 (507h6m10.721161104s) [noise=true , address=145.79.227.38:4979 , piece_id=TTBTN7CTRJU26U3B6AQKYE2KXCUXXY45AHOMBPJIOM2OGA6EZLJA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.019478927 (507h6m10.890561962s) [noise=true , address=77.90.31.13:4627 , piece_id=FAFKNLLQWAPM6EIU7LGCHWFP4I6MSJQJEVCQQSZJZFNGQND2YQ5Q, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.064226665 (507h6m10.93530971s) [noise=true , address=45.91.116.6:4973 , piece_id=KG24MG5HRPDIRSHJVCOSPFTLAW33FSZGDT723IOWBGYZ4XLWWCBQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.083232706 (507h6m10.954315751s) [noise=true , address=77.90.31.14:4663 , piece_id=KYWCYO5ZLTD3VZDLX4WGUPOIY7HLSYWL3XHRT7L4DWA2TMFVKJ4A, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.144338862 (507h6m11.015421908s) [noise=true , address=45.91.116.10:4084 , piece_id=YYBEWDUMQDXAL6K5NX3G7ZHATAXMDZJXK3SLQ456OPU7MSNV6WPQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: <nil> Dec 26 18:35:11.144365794 (507h6m11.015448839s) [seg_pos={0 0}, upload=444981]: Segment reached optimal threshold of 13 pieces. Dec 26 18:35:11.144626611 (507h6m11.015709656s) [noise=true , address=145.79.227.35:4483 , piece_id=6KKDTL7KU2QDX2IX5T6KMFWC7Z4G2EVK5R62BPEN3UFALTH2YYSQ, seg_pos={0 0}, upload=444981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 26 18:35:11.144636810 (507h6m11.015719855s) [noise=true , address=145.79.227.35:4483 , piece_id=6KKDTL7KU2QDX2IX5T6KMFWC7Z4G2EVK5R62BPEN3UFALTH2YYSQ, seg_pos={0 0}, upload=444981]: Long tail cancellation due to optimal threshold Dec 26 18:35:11.144691906 (507h6m11.015774941s) [noise=true , address=77.90.31.16:4735 , piece_id=G5DCHTYQ4YDVOIY3UHCEWMOPXOHZHCR3LCVAMJBTEYKPS5MAQLZA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: write tcp 188.166.225.170:38232->77.90.31.16:4735: use of closed network connection; write tcp 188.166.225.170:38232->77.90.31.16:4735: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38232->77.90.31.16:4735: use of closed network connection Dec 26 18:35:11.144702576 (507h6m11.015785621s) [noise=true , address=77.90.31.16:4735 , piece_id=G5DCHTYQ4YDVOIY3UHCEWMOPXOHZHCR3LCVAMJBTEYKPS5MAQLZA, seg_pos={0 0}, upload=444981]: Long tail cancellation due to optimal threshold Dec 26 18:35:11.144902136 (507h6m11.015985181s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5227ZHY2JHTRO6QS3SOY7SQEBWCEH7RVLRY6LLYT5WDYBY7P2PA, seg_pos={0 0}, upload=444981]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: expected piece hash; context canceled; EOF Dec 26 18:35:11.144912606 (507h6m11.015995661s) [noise=true , address=145.79.227.5:4222 , piece_id=Q5227ZHY2JHTRO6QS3SOY7SQEBWCEH7RVLRY6LLYT5WDYBY7P2PA, seg_pos={0 0}, upload=444981]: Long tail cancellation due to optimal threshold Dec 26 18:35:11.144939287 (507h6m11.016022332s) [seg_pos={0 0}, upload=444981]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:35:11.145035599 (507h6m11.016118664s) [upload=444981]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:35:11.145045589 (507h6m11.016128634s) [upload=444981]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:35:11.433280097 (507h6m11.304363142s) [upload=444981]: Done uploading segments... Dec 26 18:35:11.433303552 (507h6m11.304386587s) [upload=444981]: Upload finished. err: <nil> Duration: 30.002359448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X.....] duration: 6.886396358s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 6.8864291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X....] duration: 22.35005192s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:33:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 22.350080384s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X...] duration: 6.160071027s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:32:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 6.160089902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 41.499933054s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 41.499966446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X..] duration: 8.443016355s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 8.443131955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X.] duration: 29.698236632s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:31:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 29.698268424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 35.095905512s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 35.095934728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.X] duration: 30.003502052s trace: https://jaeger.dev.storj.io/trace/b06b9ed50506f0d FAIL (2025-12-26 18:30:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA) context deadline exceeded ======LOGS====== Dec 26 18:30:10.267967389 (507h1m10.139050425s) [upload=444902]: Starting upload Dec 26 18:30:10.268102978 (507h1m10.139186023s) [upload=444902]: Done starting upload Dec 26 18:30:10.268175255 (507h1m10.1392583s) [upload=444902]: Uploading segments... Dec 26 18:30:10.268312386 (507h1m10.139395432s) [upload=444902]: Got next segment. Inline: false Dec 26 18:30:10.268383913 (507h1m10.139466958s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:30:10.268390245 (507h1m10.139473291s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:30:10.519136000 (507h1m10.390219055s) [seg_pos={0 0}, upload=444902]: Begin upload segment... Dec 26 18:30:10.519192507 (507h1m10.390275552s) [seg_pos={0 0}, upload=444902]: Done begin upload segment. Dec 26 18:30:10.519226561 (507h1m10.390309607s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.519235028 (507h1m10.390318073s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520217739 (507h1m10.391300774s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520400928 (507h1m10.391483973s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.520811620 (507h1m10.391894665s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521203566 (507h1m10.392286611s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521573962 (507h1m10.392657007s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.521992619 (507h1m10.393075664s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522392450 (507h1m10.393475495s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.522960172 (507h1m10.394043207s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.523354963 (507h1m10.394438008s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525289748 (507h1m10.396372783s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.525792434 (507h1m10.396875479s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526279041 (507h1m10.397362086s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.526812617 (507h1m10.397895652s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:10.527245061 (507h1m10.398328096s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Uploading piece... Dec 26 18:30:12.144404335 (507h1m12.01548738s) [upload=444902]: Next returned nil segment Dec 26 18:30:12.144486131 (507h1m12.015569196s) [upload=444902]: Waiting for error group managing segments... Dec 26 18:30:12.527130614 (507h1m12.398213659s) [noise=true , address=145.79.227.49:4338 , piece_id=5ZIUPIA5HDGOOZCMZB5XWTX33QZROUXT6QYJK472UKD2YMWI5JNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.530644123 (507h1m12.401727168s) [noise=true , address=145.79.227.36:4517 , piece_id=A75MAPYRMZPF77WC5NZSSTDWYWVDLCOYRC2IQPPGKDLLRVGSQ4SA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.546942449 (507h1m12.418025494s) [noise=true , address=145.79.227.42:4121 , piece_id=GTHCCMTXT73UH5GK4BPYDX3A6UQV7OHN5EZKL5E32JNYQNRGSVNQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.559542481 (507h1m12.430625527s) [noise=true , address=145.79.227.14:4839 , piece_id=35ETW7ECDODE4XDORUOKNV63WP57VG4S4JT6USHMKO6W7NXZ7KMQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.574166628 (507h1m12.445249673s) [noise=true , address=145.79.227.11:4729 , piece_id=AMIX6ASBHT2KOJDKRKE3UV5ENWKGMP24ICOUUNVAZLXMOIFBYXWA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.630407832 (507h1m12.501490877s) [noise=true , address=145.79.227.2:4115 , piece_id=5KXRWOS6J5GF6N7IC76XZIBDYKZQNQ3WCLKBPUS7E75L6JRCJSNA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.761971351 (507h1m12.633054396s) [noise=true , address=45.91.116.8:4046 , piece_id=XMIB5LOCWEJ4E3WUQNOTHWRAFWLWKKOWB3J6V7HRVGTQN5ZIK2FQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.785411075 (507h1m12.65649411s) [noise=true , address=77.90.31.21:4878 , piece_id=EM4T7QLGQFCS5UCH4KT2IXTYBNDZOVGE2HCSJSZW4PW563Q7M6UA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:12.840918811 (507h1m12.712001856s) [noise=true , address=77.90.31.16:4735 , piece_id=QX6XZMGVADMH3SAXQBQVBQ2FBTXMHX7524XIM76FPVW62UFQHAJA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.467818829 (507h1m13.338901864s) [noise=true , address=45.91.116.12:4225 , piece_id=STICUCWEA4LPDCDFVTYHPP4UDRVVEG6PTX3RMB6PUBIDJQSJKH4Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.468849031 (507h1m13.339932076s) [noise=true , address=45.91.116.6:4974 , piece_id=G3KNBUMO4UTO5DHUBQGJJGHOHDOEFZ6QY4DTJ2OYMJAZ4GU6Q7WA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:13.496876309 (507h1m13.367959354s) [noise=true , address=77.90.31.14:4662 , piece_id=ZU42CUDF6S5BMYS3SMQNDCVAAHEPCVHEHMRPBVO5WXN2FFH5Z6HA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015633615 (507h1m13.88671668s) [noise=true , address=145.79.227.34:4907 , piece_id=QELA5CDKY2ETXS4GGK6GHVXNUCCPDRXCJVEP7BAMDMXYZU5SJIUA, seg_pos={0 0}, upload=444902]: Done uploading piece. err: <nil> Dec 26 18:30:14.015662509 (507h1m13.886745544s) [seg_pos={0 0}, upload=444902]: Segment reached optimal threshold of 13 pieces. Dec 26 18:30:14.015929808 (507h1m13.887012863s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015941300 (507h1m13.887024345s) [noise=true , address=45.91.116.10:4083 , piece_id=TKVUR76RI2ZMURVBFSZMJPBP2TKUKXEFRN2R644C62J2VWG572NQ, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.015954555 (507h1m13.887037611s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.015961949 (507h1m13.887045045s) [noise=true , address=145.79.227.51:4409 , piece_id=EPAYPKMULFPEB5CKAC276M4JA7UE3ILTN76WJAPTSQCFRAGZAD5Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016280536 (507h1m13.887363591s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 18:30:14.016289863 (507h1m13.887372908s) [noise=true , address=45.91.116.7:4009 , piece_id=VX5ZBAIALKKY7Z5VK2IU4OKDOAUMOK36ARCP6VFNRD4SHOVHP65Q, seg_pos={0 0}, upload=444902]: Long tail cancellation due to optimal threshold Dec 26 18:30:14.016325411 (507h1m13.887408456s) [seg_pos={0 0}, upload=444902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:30:14.016440721 (507h1m13.887523766s) [upload=444902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:30:14.016449808 (507h1m13.887532853s) [upload=444902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:30:14.318047924 (507h1m14.189130969s) [upload=444902]: Done uploading segments... Dec 26 18:30:14.318072681 (507h1m14.189155716s) [upload=444902]: Upload finished. err: <nil> Duration: 30.003519515s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 33.32335266s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:30:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.323382418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X.] duration: 23.440573065s trace: https://jaeger.dev.storj.io/trace/524d31df8f35c7ff FAIL (2025-12-26 18:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ) context deadline exceeded ======LOGS====== Dec 26 18:28:23.822347122 (506h59m23.693430157s) [upload=444874]: Starting upload Dec 26 18:28:23.822410964 (506h59m23.693493999s) [upload=444874]: Done starting upload Dec 26 18:28:23.822500815 (506h59m23.69358385s) [upload=444874]: Uploading segments... Dec 26 18:28:23.822596026 (506h59m23.693679071s) [upload=444874]: Got next segment. Inline: false Dec 26 18:28:23.822642454 (506h59m23.693725499s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:28:23.822649198 (506h59m23.693732253s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:28:24.074649680 (506h59m23.945732725s) [seg_pos={0 0}, upload=444874]: Begin upload segment... Dec 26 18:28:24.074791200 (506h59m23.945874235s) [seg_pos={0 0}, upload=444874]: Done begin upload segment. Dec 26 18:28:24.074824192 (506h59m23.945907237s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.074855793 (506h59m23.945938828s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075309056 (506h59m23.946392091s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075666957 (506h59m23.946750002s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076082839 (506h59m23.947165884s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076467922 (506h59m23.947550967s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076757614 (506h59m23.947840649s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077078766 (506h59m23.948161801s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077090618 (506h59m23.948173653s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077126346 (506h59m23.948209381s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077472445 (506h59m23.94855548s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077594388 (506h59m23.948677433s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077874671 (506h59m23.948957706s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.078020459 (506h59m23.949103494s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079200857 (506h59m23.950283902s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079589126 (506h59m23.950672171s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:25.779365970 (506h59m25.650449015s) [upload=444874]: Next returned nil segment Dec 26 18:28:25.779391659 (506h59m25.650474694s) [upload=444874]: Waiting for error group managing segments... Dec 26 18:28:26.066406545 (506h59m25.93748959s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.083768736 (506h59m25.954851771s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.090536202 (506h59m25.961619237s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.102954479 (506h59m25.974037514s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.108136156 (506h59m25.979219191s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.109335761 (506h59m25.980418796s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.114107036 (506h59m25.985190071s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.149774493 (506h59m26.020857528s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.375250659 (506h59m26.246333704s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.427833536 (506h59m26.298916571s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443659091 (506h59m26.314742136s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443997485 (506h59m26.31508053s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653512108 (506h59m26.524595143s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653537106 (506h59m26.524620151s) [seg_pos={0 0}, upload=444874]: Segment reached optimal threshold of 13 pieces. Dec 26 18:28:26.653883485 (506h59m26.524966531s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653893424 (506h59m26.524976459s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.653977345 (506h59m26.52506038s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653994046 (506h59m26.525077081s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654058188 (506h59m26.525141233s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.654067496 (506h59m26.525150531s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654096902 (506h59m26.525179937s) [seg_pos={0 0}, upload=444874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:28:26.654179278 (506h59m26.525262313s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:28:26.654187445 (506h59m26.52527048s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:28:26.972105415 (506h59m26.843188461s) [upload=444874]: Done uploading segments... Dec 26 18:28:26.972128840 (506h59m26.843211885s) [upload=444874]: Upload finished. err: <nil> Duration: 23.44060222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 15.205669173s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.205702025s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 43.599200089s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:28:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 43.599226629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.X] duration: 30.002943049s trace: https://jaeger.dev.storj.io/trace/524d31df8f35c7ff FAIL (2025-12-26 18:28:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ) context deadline exceeded ======LOGS====== Dec 26 18:28:23.822347122 (506h59m23.693430157s) [upload=444874]: Starting upload Dec 26 18:28:23.822410964 (506h59m23.693493999s) [upload=444874]: Done starting upload Dec 26 18:28:23.822500815 (506h59m23.69358385s) [upload=444874]: Uploading segments... Dec 26 18:28:23.822596026 (506h59m23.693679071s) [upload=444874]: Got next segment. Inline: false Dec 26 18:28:23.822642454 (506h59m23.693725499s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:28:23.822649198 (506h59m23.693732253s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:28:24.074649680 (506h59m23.945732725s) [seg_pos={0 0}, upload=444874]: Begin upload segment... Dec 26 18:28:24.074791200 (506h59m23.945874235s) [seg_pos={0 0}, upload=444874]: Done begin upload segment. Dec 26 18:28:24.074824192 (506h59m23.945907237s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.074855793 (506h59m23.945938828s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075309056 (506h59m23.946392091s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.075666957 (506h59m23.946750002s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076082839 (506h59m23.947165884s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076467922 (506h59m23.947550967s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.076757614 (506h59m23.947840649s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077078766 (506h59m23.948161801s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077090618 (506h59m23.948173653s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077126346 (506h59m23.948209381s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077472445 (506h59m23.94855548s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077594388 (506h59m23.948677433s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.077874671 (506h59m23.948957706s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.078020459 (506h59m23.949103494s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079200857 (506h59m23.950283902s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:24.079589126 (506h59m23.950672171s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Uploading piece... Dec 26 18:28:25.779365970 (506h59m25.650449015s) [upload=444874]: Next returned nil segment Dec 26 18:28:25.779391659 (506h59m25.650474694s) [upload=444874]: Waiting for error group managing segments... Dec 26 18:28:26.066406545 (506h59m25.93748959s) [noise=true , address=145.79.227.10:4401 , piece_id=SI3LHI5BPBS3PAJB74FGPESAKVO2IFGE6V4UVZZ7PLMFK3VRXNKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.083768736 (506h59m25.954851771s) [noise=true , address=145.79.227.46:4603 , piece_id=ZHFEEQ6NOHP4TD5IA5GKTUPW76POCWCOQCSM33WT5JDARRSTKRKQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.090536202 (506h59m25.961619237s) [noise=true , address=145.79.227.45:4231 , piece_id=T6BOS3MCNPAMPZH3N6BSQNCSQ2R3WA4HKT563MCRVPJGBTYZJLUQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.102954479 (506h59m25.974037514s) [noise=true , address=145.79.227.3:4149 , piece_id=DHDHV36CYF56CDUNSKLNZL37JD3MKY57LMCYIRZCAW46AFYMU3VA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.108136156 (506h59m25.979219191s) [noise=true , address=145.79.227.38:4979 , piece_id=DLV3K2DP7ASFVJ4XM2B24P5PXVE74OQ5CDT6CUQFBIZIZWNDKC2Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.109335761 (506h59m25.980418796s) [noise=true , address=145.79.227.13:4804 , piece_id=ZZKEBY64EAYOQUNXITN34OTKOVKVU265X5UPZQEKREBWJNNRPBPQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.114107036 (506h59m25.985190071s) [noise=true , address=145.79.227.52:4445 , piece_id=BDIDHUSDZAQIFKEEN2S5MQLF3I6CD66LQ6HFILT53ODN4KXGYD7Q, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.149774493 (506h59m26.020857528s) [noise=true , address=145.79.227.14:4839 , piece_id=3F5RR2FP2BGTP56WJBVEN2X4Q52ZSXX7AZFEDUW66JBFPHLPA6NA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.375250659 (506h59m26.246333704s) [noise=true , address=45.91.116.8:4047 , piece_id=RVRJW5JPVMHSR5Y7RN6XQETQLA6LV726S4A5GYAWX2DC5DNUVDBQ, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.427833536 (506h59m26.298916571s) [noise=true , address=45.91.116.14:4189 , piece_id=YAWBY3SI2SIZXGFL5RAQW5DY4HLQENKC366QNXRJKGNPLFPTLBRA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443659091 (506h59m26.314742136s) [noise=true , address=45.91.116.13:4154 , piece_id=RJ2L576LCPQTSWW4UEBIN5VOUXJRTPQJ7PISMGAGM4MOMMTW37KA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.443997485 (506h59m26.31508053s) [noise=true , address=145.79.227.16:4911 , piece_id=HC7BIPT556S6M3UNOKEPW2RYYCK5UI2DAC4ZMFKGXAURPLGASRQA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653512108 (506h59m26.524595143s) [noise=true , address=45.91.116.5:4300 , piece_id=VQJFFUVDIPOSJIYDS5VVV6IJKRUPHHFROUXPG7RWPD7XIYOHC2YA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: <nil> Dec 26 18:28:26.653537106 (506h59m26.524620151s) [seg_pos={0 0}, upload=444874]: Segment reached optimal threshold of 13 pieces. Dec 26 18:28:26.653883485 (506h59m26.524966531s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653893424 (506h59m26.524976459s) [noise=true , address=145.79.227.33:4871 , piece_id=4DQKLIJIMMHFO2W5TYL7I2DY5I3Z323EMBXYWVI4YWEHDW23ZC6A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.653977345 (506h59m26.52506038s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.653994046 (506h59m26.525077081s) [noise=true , address=145.79.227.40:4050 , piece_id=VFX5GL57OCE7CFOFTME7JTQQEDXM4EK3PJFUWLFKVYLZTNENQGKA, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654058188 (506h59m26.525141233s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 26 18:28:26.654067496 (506h59m26.525150531s) [noise=true , address=45.91.116.4:4940 , piece_id=MZDXJ7XZXICAC5NHIQ5UOGCQBJDOCPM2WFGGTCKIU43ITHOQP35A, seg_pos={0 0}, upload=444874]: Long tail cancellation due to optimal threshold Dec 26 18:28:26.654096902 (506h59m26.525179937s) [seg_pos={0 0}, upload=444874]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:28:26.654179278 (506h59m26.525262313s) [upload=444874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:28:26.654187445 (506h59m26.52527048s) [upload=444874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:28:26.972105415 (506h59m26.843188461s) [upload=444874]: Done uploading segments... Dec 26 18:28:26.972128840 (506h59m26.843211885s) [upload=444874]: Upload finished. err: <nil> Duration: 30.002968508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XX.X.XXX.........X..X.....X...XX.........X.....X..X..X.] duration: 16.154629659s trace: https://jaeger.dev.storj.io/trace/3e598be3dee99e8d FAIL (2025-12-26 18:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:27:04.814038872 (506h58m4.685121917s) [upload=444856]: Starting upload Dec 26 18:27:04.814093786 (506h58m4.685176822s) [upload=444856]: Done starting upload Dec 26 18:27:04.814313374 (506h58m4.685396409s) [upload=444856]: Uploading segments... Dec 26 18:27:04.814362368 (506h58m4.685445403s) [upload=444856]: Got next segment. Inline: false Dec 26 18:27:04.814388688 (506h58m4.685471723s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:27:04.814392967 (506h58m4.685476012s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:27:05.070533924 (506h58m4.941616959s) [seg_pos={0 0}, upload=444856]: Begin upload segment... Dec 26 18:27:05.070583608 (506h58m4.941666653s) [seg_pos={0 0}, upload=444856]: Done begin upload segment. Dec 26 18:27:05.070611822 (506h58m4.941694857s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.070664092 (506h58m4.941747137s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071177449 (506h58m4.942260494s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071661871 (506h58m4.942744907s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072017379 (506h58m4.943100414s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072301139 (506h58m4.943384184s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072582205 (506h58m4.94366525s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072878969 (506h58m4.943962014s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073248253 (506h58m4.944331288s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073591316 (506h58m4.944674361s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075023643 (506h58m4.946106688s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075078929 (506h58m4.946161974s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.076675308 (506h58m4.947758373s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.077105327 (506h58m4.948188362s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.080306162 (506h58m4.951389207s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.083793563 (506h58m4.954876608s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:06.692846281 (506h58m6.563929326s) [upload=444856]: Next returned nil segment Dec 26 18:27:06.692883091 (506h58m6.563966136s) [upload=444856]: Waiting for error group managing segments... Dec 26 18:27:07.079343870 (506h58m6.950426905s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.080151929 (506h58m6.951234964s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.083307027 (506h58m6.954390072s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.087252359 (506h58m6.958335394s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.090869848 (506h58m6.961952893s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.101038851 (506h58m6.972121896s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.112619303 (506h58m6.983702348s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.114945773 (506h58m6.986028808s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.122221778 (506h58m6.993304823s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.361307158 (506h58m7.232390193s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.472087023 (506h58m7.343170068s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.481307370 (506h58m7.352390415s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096007195 (506h58m7.96709024s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096032524 (506h58m7.967115559s) [seg_pos={0 0}, upload=444856]: Segment reached optimal threshold of 13 pieces. Dec 26 18:27:08.096286228 (506h58m7.967369263s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096294854 (506h58m7.967377889s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096557273 (506h58m7.967640308s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096565530 (506h58m7.967648575s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096738468 (506h58m7.967821523s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection Dec 26 18:27:08.096757885 (506h58m7.96784093s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096785638 (506h58m7.967868673s) [seg_pos={0 0}, upload=444856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:27:08.096877743 (506h58m7.967960788s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:27:08.096884136 (506h58m7.967967171s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:27:08.398825365 (506h58m8.26990841s) [upload=444856]: Done uploading segments... Dec 26 18:27:08.398846124 (506h58m8.269929159s) [upload=444856]: Upload finished. err: <nil> Duration: 16.154656349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 28.774871694s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:27:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 28.77489012s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.....XX.X.XXX.........X..X.....X...XX.........X.....X..X..X] duration: 30.002862384s trace: https://jaeger.dev.storj.io/trace/3e598be3dee99e8d FAIL (2025-12-26 18:27:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 18:27:04.814038872 (506h58m4.685121917s) [upload=444856]: Starting upload Dec 26 18:27:04.814093786 (506h58m4.685176822s) [upload=444856]: Done starting upload Dec 26 18:27:04.814313374 (506h58m4.685396409s) [upload=444856]: Uploading segments... Dec 26 18:27:04.814362368 (506h58m4.685445403s) [upload=444856]: Got next segment. Inline: false Dec 26 18:27:04.814388688 (506h58m4.685471723s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:27:04.814392967 (506h58m4.685476012s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:27:05.070533924 (506h58m4.941616959s) [seg_pos={0 0}, upload=444856]: Begin upload segment... Dec 26 18:27:05.070583608 (506h58m4.941666653s) [seg_pos={0 0}, upload=444856]: Done begin upload segment. Dec 26 18:27:05.070611822 (506h58m4.941694857s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.070664092 (506h58m4.941747137s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071177449 (506h58m4.942260494s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.071661871 (506h58m4.942744907s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072017379 (506h58m4.943100414s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072301139 (506h58m4.943384184s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072582205 (506h58m4.94366525s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.072878969 (506h58m4.943962014s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073248253 (506h58m4.944331288s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.073591316 (506h58m4.944674361s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075023643 (506h58m4.946106688s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.075078929 (506h58m4.946161974s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.076675308 (506h58m4.947758373s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.077105327 (506h58m4.948188362s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.080306162 (506h58m4.951389207s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:05.083793563 (506h58m4.954876608s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Uploading piece... Dec 26 18:27:06.692846281 (506h58m6.563929326s) [upload=444856]: Next returned nil segment Dec 26 18:27:06.692883091 (506h58m6.563966136s) [upload=444856]: Waiting for error group managing segments... Dec 26 18:27:07.079343870 (506h58m6.950426905s) [noise=true , address=145.79.227.11:4729 , piece_id=QQ3G423RM3SS2QHZZBOLKQ6GN3ZR5QJPNEKOZIEGTVLUEPNHOAJQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.080151929 (506h58m6.951234964s) [noise=true , address=145.79.227.36:4517 , piece_id=SOR4UXLCDUFWIPG7EQ753EO2ESIVUWWHMXL3VDXTB42N7J67VLHQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.083307027 (506h58m6.954390072s) [noise=true , address=145.79.227.39:4015 , piece_id=4TKQLJUJR2GRJCNPS7LZTNE5JKFWCARETVJ4RRELIC7GIYG7KKCQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.087252359 (506h58m6.958335394s) [noise=true , address=145.79.227.21:4475 , piece_id=SGRYERO6Q6DEFR7LZSFNWSTEFRB3Y3EI2HMBBABXKH2WHW7ZJ24A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.090869848 (506h58m6.961952893s) [noise=true , address=145.79.227.32:4836 , piece_id=XF7VQ7Q4DV42AGMN74RVKPY26QVFX3FVH726Z55CLGQ7MTTVFBUA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.101038851 (506h58m6.972121896s) [noise=true , address=145.79.227.22:4511 , piece_id=ZWZBJE6QZY6C4NP6X75MOX2KHLPXMK2SIHX2Z235PAM3BG45Z7OQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.112619303 (506h58m6.983702348s) [noise=true , address=145.79.227.48:4303 , piece_id=XJHI2LYEK62QFIIYKMCIONNYHBVJKGEBMYPJZZDQN2QLOUL66Q6A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.114945773 (506h58m6.986028808s) [noise=true , address=145.79.227.10:4401 , piece_id=7FNANWRWX4ASK6LSH3EVYAMCAMIE7UAKBBNRY4TIFNPK63TROQMQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.122221778 (506h58m6.993304823s) [noise=true , address=145.79.227.14:4839 , piece_id=ICNQQGVL4YX7MBUQUUYTIEFYD7BSXZ3JGKEI23IIE3PYABP5EO7A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.361307158 (506h58m7.232390193s) [noise=true , address=45.91.116.9:4264 , piece_id=OGUTLNL4VVGZZ3QGW4PBWC2KXLZONWSPM6GDB3NMP4MWA5ALEBRA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.472087023 (506h58m7.343170068s) [noise=true , address=45.91.116.5:4299 , piece_id=WRQSHAG5ZAIVPYF3ZN2QIEIM4B5XGBDPLQGL52JI7XV77P3SM7OA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:07.481307370 (506h58m7.352390415s) [noise=true , address=45.91.116.13:4156 , piece_id=WVDRVEBPKNGPRZ3S4W6YFZNCEUZXN54MHEZ3Y6BXFMSLKI3J4TTQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096007195 (506h58m7.96709024s) [noise=true , address=77.90.31.18:4808 , piece_id=4GDMSXXA4GAHTKRZGMOOTFQ65E2Z2PJSVWC7KKNYQ6E66F2LVTQA, seg_pos={0 0}, upload=444856]: Done uploading piece. err: <nil> Dec 26 18:27:08.096032524 (506h58m7.967115559s) [seg_pos={0 0}, upload=444856]: Segment reached optimal threshold of 13 pieces. Dec 26 18:27:08.096286228 (506h58m7.967369263s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096294854 (506h58m7.967377889s) [noise=true , address=145.79.227.43:4158 , piece_id=5NKCRIQULIWTV2K3YYD5RPLDAWUPJQ34OACLRDAAJOJVH6BZSN2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096557273 (506h58m7.967640308s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 26 18:27:08.096565530 (506h58m7.967648575s) [noise=true , address=77.90.31.16:4736 , piece_id=4YV4G6OUIWQDOAIQE7PYHAH7GFIPWQVFPPEZ5HXTXXNFQAF35S2A, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096738468 (506h58m7.967821523s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45402->77.90.31.21:4880: use of closed network connection Dec 26 18:27:08.096757885 (506h58m7.96784093s) [noise=true , address=77.90.31.21:4880 , piece_id=DMXNEEWI6V5YRU5YGIAM26PXLG3GFP5OMAU4NTOGOZRGSUKWLTLQ, seg_pos={0 0}, upload=444856]: Long tail cancellation due to optimal threshold Dec 26 18:27:08.096785638 (506h58m7.967868673s) [seg_pos={0 0}, upload=444856]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:27:08.096877743 (506h58m7.967960788s) [upload=444856]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:27:08.096884136 (506h58m7.967967171s) [upload=444856]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:27:08.398825365 (506h58m8.26990841s) [upload=444856]: Done uploading segments... Dec 26 18:27:08.398846124 (506h58m8.269929159s) [upload=444856]: Upload finished. err: <nil> Duration: 30.00288679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 34.515609263s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:27:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 34.515639021s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX.....XX.X.XXX.........X..X.....X...XX.........X.....X..X..] duration: 9.17548808s trace: https://jaeger.dev.storj.io/trace/4d484ce3fc14ae0 FAIL (2025-12-26 18:26:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ) context deadline exceeded ======LOGS====== 6 18:25:19.666927494 (506h56m19.538010539s) [upload=444830]: Starting upload Dec 26 18:25:19.667038895 (506h56m19.538121961s) [upload=444830]: Done starting upload Dec 26 18:25:19.667174374 (506h56m19.538257409s) [upload=444830]: Uploading segments... Dec 26 18:25:19.667258674 (506h56m19.538341719s) [upload=444830]: Got next segment. Inline: false Dec 26 18:25:19.667296165 (506h56m19.5383792s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:25:19.667301856 (506h56m19.538384901s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:25:19.918625050 (506h56m19.789708105s) [seg_pos={0 0}, upload=444830]: Begin upload segment... Dec 26 18:25:19.918761218 (506h56m19.789844253s) [seg_pos={0 0}, upload=444830]: Done begin upload segment. Dec 26 18:25:19.918834739 (506h56m19.789917784s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.918928477 (506h56m19.790011522s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919612670 (506h56m19.790695705s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919839992 (506h56m19.790923037s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919995649 (506h56m19.791078694s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920235296 (506h56m19.791318341s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920356225 (506h56m19.79143927s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920575754 (506h56m19.791658799s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920713106 (506h56m19.791796151s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920967009 (506h56m19.792050054s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921151460 (506h56m19.792234495s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921344848 (506h56m19.792427884s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921570267 (506h56m19.792653312s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921714832 (506h56m19.792797877s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922067303 (506h56m19.793150348s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922212541 (506h56m19.793295576s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:21.625781415 (506h56m21.49686449s) [upload=444830]: Next returned nil segment Dec 26 18:25:21.625838002 (506h56m21.496921047s) [upload=444830]: Waiting for error group managing segments... Dec 26 18:25:21.919637218 (506h56m21.790720253s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.920253681 (506h56m21.791336727s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.929953431 (506h56m21.801036466s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.934488106 (506h56m21.805571141s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.943745083 (506h56m21.814828128s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.946938384 (506h56m21.818021429s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.950590989 (506h56m21.821674024s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.952817939 (506h56m21.823900974s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.954184631 (506h56m21.825267677s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.092833324 (506h56m21.963916359s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.291735528 (506h56m22.162818583s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.364544489 (506h56m22.235627534s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842766488 (506h56m22.713849533s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842794591 (506h56m22.713877636s) [seg_pos={0 0}, upload=444830]: Segment reached optimal threshold of 13 pieces. Dec 26 18:25:22.843025661 (506h56m22.714108706s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843036923 (506h56m22.714119968s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843137895 (506h56m22.71422094s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843160578 (506h56m22.714243623s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843279695 (506h56m22.71436274s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection Dec 26 18:25:22.843293862 (506h56m22.714376907s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843317407 (506h56m22.714400452s) [seg_pos={0 0}, upload=444830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:25:22.843415354 (506h56m22.714498399s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:25:22.843422667 (506h56m22.714505712s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:25:23.146348141 (506h56m23.017431176s) [upload=444830]: Done uploading segments... Dec 26 18:25:23.146369531 (506h56m23.017452576s) [upload=444830]: Upload finished. err: <nil> Duration: 9.175557062s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X..X.] duration: 19.483778334s trace: https://jaeger.dev.storj.io/trace/4d484ce3fc14ae0 FAIL (2025-12-26 18:26:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ) context deadline exceeded ======LOGS====== 6 18:25:19.666927494 (506h56m19.538010539s) [upload=444830]: Starting upload Dec 26 18:25:19.667038895 (506h56m19.538121961s) [upload=444830]: Done starting upload Dec 26 18:25:19.667174374 (506h56m19.538257409s) [upload=444830]: Uploading segments... Dec 26 18:25:19.667258674 (506h56m19.538341719s) [upload=444830]: Got next segment. Inline: false Dec 26 18:25:19.667296165 (506h56m19.5383792s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:25:19.667301856 (506h56m19.538384901s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:25:19.918625050 (506h56m19.789708105s) [seg_pos={0 0}, upload=444830]: Begin upload segment... Dec 26 18:25:19.918761218 (506h56m19.789844253s) [seg_pos={0 0}, upload=444830]: Done begin upload segment. Dec 26 18:25:19.918834739 (506h56m19.789917784s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.918928477 (506h56m19.790011522s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919612670 (506h56m19.790695705s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919839992 (506h56m19.790923037s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919995649 (506h56m19.791078694s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920235296 (506h56m19.791318341s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920356225 (506h56m19.79143927s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920575754 (506h56m19.791658799s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920713106 (506h56m19.791796151s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920967009 (506h56m19.792050054s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921151460 (506h56m19.792234495s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921344848 (506h56m19.792427884s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921570267 (506h56m19.792653312s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921714832 (506h56m19.792797877s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922067303 (506h56m19.793150348s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922212541 (506h56m19.793295576s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:21.625781415 (506h56m21.49686449s) [upload=444830]: Next returned nil segment Dec 26 18:25:21.625838002 (506h56m21.496921047s) [upload=444830]: Waiting for error group managing segments... Dec 26 18:25:21.919637218 (506h56m21.790720253s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.920253681 (506h56m21.791336727s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.929953431 (506h56m21.801036466s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.934488106 (506h56m21.805571141s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.943745083 (506h56m21.814828128s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.946938384 (506h56m21.818021429s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.950590989 (506h56m21.821674024s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.952817939 (506h56m21.823900974s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.954184631 (506h56m21.825267677s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.092833324 (506h56m21.963916359s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.291735528 (506h56m22.162818583s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.364544489 (506h56m22.235627534s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842766488 (506h56m22.713849533s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842794591 (506h56m22.713877636s) [seg_pos={0 0}, upload=444830]: Segment reached optimal threshold of 13 pieces. Dec 26 18:25:22.843025661 (506h56m22.714108706s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843036923 (506h56m22.714119968s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843137895 (506h56m22.71422094s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843160578 (506h56m22.714243623s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843279695 (506h56m22.71436274s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection Dec 26 18:25:22.843293862 (506h56m22.714376907s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843317407 (506h56m22.714400452s) [seg_pos={0 0}, upload=444830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:25:22.843415354 (506h56m22.714498399s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:25:22.843422667 (506h56m22.714505712s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:25:23.146348141 (506h56m23.017431176s) [upload=444830]: Done uploading segments... Dec 26 18:25:23.146369531 (506h56m23.017452576s) [upload=444830]: Upload finished. err: <nil> Duration: 19.483872503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X..X] duration: 30.004232502s trace: https://jaeger.dev.storj.io/trace/4d484ce3fc14ae0 FAIL (2025-12-26 18:25:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ) context deadline exceeded ======LOGS====== 6 18:25:19.666927494 (506h56m19.538010539s) [upload=444830]: Starting upload Dec 26 18:25:19.667038895 (506h56m19.538121961s) [upload=444830]: Done starting upload Dec 26 18:25:19.667174374 (506h56m19.538257409s) [upload=444830]: Uploading segments... Dec 26 18:25:19.667258674 (506h56m19.538341719s) [upload=444830]: Got next segment. Inline: false Dec 26 18:25:19.667296165 (506h56m19.5383792s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:25:19.667301856 (506h56m19.538384901s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:25:19.918625050 (506h56m19.789708105s) [seg_pos={0 0}, upload=444830]: Begin upload segment... Dec 26 18:25:19.918761218 (506h56m19.789844253s) [seg_pos={0 0}, upload=444830]: Done begin upload segment. Dec 26 18:25:19.918834739 (506h56m19.789917784s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.918928477 (506h56m19.790011522s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919612670 (506h56m19.790695705s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919839992 (506h56m19.790923037s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.919995649 (506h56m19.791078694s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920235296 (506h56m19.791318341s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920356225 (506h56m19.79143927s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920575754 (506h56m19.791658799s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920713106 (506h56m19.791796151s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.920967009 (506h56m19.792050054s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921151460 (506h56m19.792234495s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921344848 (506h56m19.792427884s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921570267 (506h56m19.792653312s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.921714832 (506h56m19.792797877s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922067303 (506h56m19.793150348s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:19.922212541 (506h56m19.793295576s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Uploading piece... Dec 26 18:25:21.625781415 (506h56m21.49686449s) [upload=444830]: Next returned nil segment Dec 26 18:25:21.625838002 (506h56m21.496921047s) [upload=444830]: Waiting for error group managing segments... Dec 26 18:25:21.919637218 (506h56m21.790720253s) [noise=true , address=145.79.227.51:4409 , piece_id=D2G7AM2VNS77FUDFFAZ5IJDI3HGNJYMKQUK4Z5JUBL3VSFRT4ZLQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.920253681 (506h56m21.791336727s) [noise=true , address=145.79.227.39:4013 , piece_id=CSM5J3CPJ4AYOKHSVLBYIM6T7VGFRGMKBAH7BYEKJWHYLBJ4FMFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.929953431 (506h56m21.801036466s) [noise=true , address=145.79.227.22:4511 , piece_id=BMFAFUOM3HZAXCDJV2RB3QPI453KK23HPM6FDJJCF3ELECANLVFA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.934488106 (506h56m21.805571141s) [noise=true , address=145.79.227.21:4474 , piece_id=46DB3FQKA33EDW7GXVXMSDPI3ZEOUR3ADP3WGLRT4X4GXKBN2H7Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.943745083 (506h56m21.814828128s) [noise=true , address=145.79.227.37:4565 , piece_id=3VJMU5PIWEXDUBP6KGY3JK5J5HYWZ64XZFSZNXL2PPYAE2NWK4OQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.946938384 (506h56m21.818021429s) [noise=true , address=145.79.227.42:4122 , piece_id=ULH3XHAYVXRIQKOTGUTUYIU5M4T2XGQ7UJLKDW4PGPUIDSLQP4IA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.950590989 (506h56m21.821674024s) [noise=true , address=145.79.227.16:4909 , piece_id=TUITUPXYBG43YVFXHDW7LKNTIQWHBDXBM3E3D75PPZ5OVGVTC76Q, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.952817939 (506h56m21.823900974s) [noise=true , address=145.79.227.49:4339 , piece_id=5BRHLKVAJ2S7LBH5CH5SE3YJN5LSX4IM2IYWZWIUOTF73LMCWPMA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:21.954184631 (506h56m21.825267677s) [noise=true , address=145.79.227.47:4267 , piece_id=F6AGT52QXC2SOWGU5XOOG4PJK2OCCCFUU3OMLCV5E2FIJ45SATGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.092833324 (506h56m21.963916359s) [noise=true , address=77.90.31.18:4808 , piece_id=XOW4J4T67WQW7HUORTKM2MHDYLJKBZYBXOJ5XNYLQEJJTEFLWLPA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.291735528 (506h56m22.162818583s) [noise=true , address=45.91.116.4:4937 , piece_id=DOM3EPX5HZBZR3VGVEO5WNPI5H7YDOBTNKTEPX5IA5GW5YBKHF2A, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.364544489 (506h56m22.235627534s) [noise=true , address=45.91.116.7:4012 , piece_id=VKGGWIBE6RX7AFJUBKJ2MSAEABXCUQQG3X36JWTCR6UG73WXVBWQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842766488 (506h56m22.713849533s) [noise=true , address=45.91.116.5:4300 , piece_id=CTQ5I3FGJGLOFFSWZC6ZNPYI5RFZBQJTZ2DTDMY4CXTGNCCEZEMQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: <nil> Dec 26 18:25:22.842794591 (506h56m22.713877636s) [seg_pos={0 0}, upload=444830]: Segment reached optimal threshold of 13 pieces. Dec 26 18:25:22.843025661 (506h56m22.714108706s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843036923 (506h56m22.714119968s) [noise=true , address=145.79.227.3:4151 , piece_id=5H6T6Q3AVDFBA3IE7NNZ7JD5PGSFWLAURSET2HV2EDPKLQWYFHDA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843137895 (506h56m22.71422094s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: expected piece hash; context canceled; EOF Dec 26 18:25:22.843160578 (506h56m22.714243623s) [noise=true , address=145.79.227.19:4077 , piece_id=EVVLHQGGWORCGY4Z2ZS5MBKWTHJNK7ASJXCSGDG4SLGIXFBGTVGQ, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843279695 (506h56m22.71436274s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39562->145.79.227.5:4221: use of closed network connection Dec 26 18:25:22.843293862 (506h56m22.714376907s) [noise=true , address=145.79.227.5:4221 , piece_id=H5GK6DCSW55XEWO4IPOG5RVDXLDDNSBAZ3MEUMJ7E3YNVQIAXOKA, seg_pos={0 0}, upload=444830]: Long tail cancellation due to optimal threshold Dec 26 18:25:22.843317407 (506h56m22.714400452s) [seg_pos={0 0}, upload=444830]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:25:22.843415354 (506h56m22.714498399s) [upload=444830]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:25:22.843422667 (506h56m22.714505712s) [upload=444830]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:25:23.146348141 (506h56m23.017431176s) [upload=444830]: Done uploading segments... Dec 26 18:25:23.146369531 (506h56m23.017452576s) [upload=444830]: Upload finished. err: <nil> Duration: 30.004256046s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.807805469s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 5.807834505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 34.948449578s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:25:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 34.94848826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X..] duration: 6.524947205s trace: https://jaeger.dev.storj.io/trace/46a03f883651bcf3 FAIL (2025-12-26 18:24:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ) context deadline exceeded ======LOGS====== 18:22:20.052924283 (506h53m19.924007328s) [upload=444786]: Starting upload Dec 26 18:22:20.053003855 (506h53m19.92408692s) [upload=444786]: Done starting upload Dec 26 18:22:20.053185570 (506h53m19.924268615s) [upload=444786]: Uploading segments... Dec 26 18:22:20.053283848 (506h53m19.924366983s) [upload=444786]: Got next segment. Inline: false Dec 26 18:22:20.053328452 (506h53m19.924411537s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:22:20.053338251 (506h53m19.924421296s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:22:20.300773130 (506h53m20.171856186s) [seg_pos={0 0}, upload=444786]: Begin upload segment... Dec 26 18:22:20.300829749 (506h53m20.171912794s) [seg_pos={0 0}, upload=444786]: Done begin upload segment. Dec 26 18:22:20.300860326 (506h53m20.171943361s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.301243986 (506h53m20.172327052s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302210267 (506h53m20.173293302s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302538793 (506h53m20.173621838s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302911883 (506h53m20.173994918s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303243464 (506h53m20.174326499s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303608820 (506h53m20.174691855s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303906076 (506h53m20.174989121s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304052885 (506h53m20.17513592s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304543570 (506h53m20.175626605s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305253973 (506h53m20.176337018s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305734778 (506h53m20.176817833s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306250310 (506h53m20.177333355s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306729412 (506h53m20.177812447s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.307243411 (506h53m20.178326456s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.308077850 (506h53m20.179160895s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:22.021189354 (506h53m21.892272399s) [upload=444786]: Next returned nil segment Dec 26 18:22:22.021203520 (506h53m21.892286566s) [upload=444786]: Waiting for error group managing segments... Dec 26 18:22:22.306790919 (506h53m22.177873964s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.317440578 (506h53m22.188523623s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.324498949 (506h53m22.195581994s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.332376839 (506h53m22.203459874s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.335008621 (506h53m22.206091666s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.337891891 (506h53m22.208974926s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.338036998 (506h53m22.209120033s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.357245244 (506h53m22.228328289s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.491011368 (506h53m22.362094403s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.492463102 (506h53m22.363546137s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.543901169 (506h53m22.414984204s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.555202418 (506h53m22.426285454s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584924595 (506h53m22.45600763s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584944242 (506h53m22.456027277s) [seg_pos={0 0}, upload=444786]: Segment reached optimal threshold of 13 pieces. Dec 26 18:22:22.585279571 (506h53m22.456362616s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585291072 (506h53m22.456374107s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585390301 (506h53m22.456473366s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection Dec 26 18:22:22.585420619 (506h53m22.456503674s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585629497 (506h53m22.456712552s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585657961 (506h53m22.456741006s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585679984 (506h53m22.456763029s) [seg_pos={0 0}, upload=444786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:22:22.585785955 (506h53m22.45686899s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:22:22.585792888 (506h53m22.456875923s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:22:22.894574533 (506h53m22.765657578s) [upload=444786]: Done uploading segments... Dec 26 18:22:22.894598457 (506h53m22.765681492s) [upload=444786]: Upload finished. err: <nil> Duration: 6.524976601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 20.190971592s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 20.191000588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 37.415631616s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 37.415661073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X.] duration: 6.213801429s trace: https://jaeger.dev.storj.io/trace/46a03f883651bcf3 FAIL (2025-12-26 18:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ) context deadline exceeded ======LOGS====== 18:22:20.052924283 (506h53m19.924007328s) [upload=444786]: Starting upload Dec 26 18:22:20.053003855 (506h53m19.92408692s) [upload=444786]: Done starting upload Dec 26 18:22:20.053185570 (506h53m19.924268615s) [upload=444786]: Uploading segments... Dec 26 18:22:20.053283848 (506h53m19.924366983s) [upload=444786]: Got next segment. Inline: false Dec 26 18:22:20.053328452 (506h53m19.924411537s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:22:20.053338251 (506h53m19.924421296s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:22:20.300773130 (506h53m20.171856186s) [seg_pos={0 0}, upload=444786]: Begin upload segment... Dec 26 18:22:20.300829749 (506h53m20.171912794s) [seg_pos={0 0}, upload=444786]: Done begin upload segment. Dec 26 18:22:20.300860326 (506h53m20.171943361s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.301243986 (506h53m20.172327052s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302210267 (506h53m20.173293302s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302538793 (506h53m20.173621838s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302911883 (506h53m20.173994918s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303243464 (506h53m20.174326499s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303608820 (506h53m20.174691855s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303906076 (506h53m20.174989121s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304052885 (506h53m20.17513592s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304543570 (506h53m20.175626605s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305253973 (506h53m20.176337018s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305734778 (506h53m20.176817833s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306250310 (506h53m20.177333355s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306729412 (506h53m20.177812447s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.307243411 (506h53m20.178326456s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.308077850 (506h53m20.179160895s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:22.021189354 (506h53m21.892272399s) [upload=444786]: Next returned nil segment Dec 26 18:22:22.021203520 (506h53m21.892286566s) [upload=444786]: Waiting for error group managing segments... Dec 26 18:22:22.306790919 (506h53m22.177873964s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.317440578 (506h53m22.188523623s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.324498949 (506h53m22.195581994s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.332376839 (506h53m22.203459874s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.335008621 (506h53m22.206091666s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.337891891 (506h53m22.208974926s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.338036998 (506h53m22.209120033s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.357245244 (506h53m22.228328289s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.491011368 (506h53m22.362094403s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.492463102 (506h53m22.363546137s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.543901169 (506h53m22.414984204s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.555202418 (506h53m22.426285454s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584924595 (506h53m22.45600763s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584944242 (506h53m22.456027277s) [seg_pos={0 0}, upload=444786]: Segment reached optimal threshold of 13 pieces. Dec 26 18:22:22.585279571 (506h53m22.456362616s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585291072 (506h53m22.456374107s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585390301 (506h53m22.456473366s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection Dec 26 18:22:22.585420619 (506h53m22.456503674s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585629497 (506h53m22.456712552s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585657961 (506h53m22.456741006s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585679984 (506h53m22.456763029s) [seg_pos={0 0}, upload=444786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:22:22.585785955 (506h53m22.45686899s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:22:22.585792888 (506h53m22.456875923s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:22:22.894574533 (506h53m22.765657578s) [upload=444786]: Done uploading segments... Dec 26 18:22:22.894598457 (506h53m22.765681492s) [upload=444786]: Upload finished. err: <nil> Duration: 6.21382876s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 48.093075117s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:23:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 48.09310374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....X] duration: 30.003734751s trace: https://jaeger.dev.storj.io/trace/46a03f883651bcf3 FAIL (2025-12-26 18:22:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ) context deadline exceeded ======LOGS====== 18:22:20.052924283 (506h53m19.924007328s) [upload=444786]: Starting upload Dec 26 18:22:20.053003855 (506h53m19.92408692s) [upload=444786]: Done starting upload Dec 26 18:22:20.053185570 (506h53m19.924268615s) [upload=444786]: Uploading segments... Dec 26 18:22:20.053283848 (506h53m19.924366983s) [upload=444786]: Got next segment. Inline: false Dec 26 18:22:20.053328452 (506h53m19.924411537s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:22:20.053338251 (506h53m19.924421296s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:22:20.300773130 (506h53m20.171856186s) [seg_pos={0 0}, upload=444786]: Begin upload segment... Dec 26 18:22:20.300829749 (506h53m20.171912794s) [seg_pos={0 0}, upload=444786]: Done begin upload segment. Dec 26 18:22:20.300860326 (506h53m20.171943361s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.301243986 (506h53m20.172327052s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302210267 (506h53m20.173293302s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302538793 (506h53m20.173621838s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.302911883 (506h53m20.173994918s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303243464 (506h53m20.174326499s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303608820 (506h53m20.174691855s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.303906076 (506h53m20.174989121s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304052885 (506h53m20.17513592s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.304543570 (506h53m20.175626605s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305253973 (506h53m20.176337018s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.305734778 (506h53m20.176817833s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306250310 (506h53m20.177333355s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.306729412 (506h53m20.177812447s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.307243411 (506h53m20.178326456s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:20.308077850 (506h53m20.179160895s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Uploading piece... Dec 26 18:22:22.021189354 (506h53m21.892272399s) [upload=444786]: Next returned nil segment Dec 26 18:22:22.021203520 (506h53m21.892286566s) [upload=444786]: Waiting for error group managing segments... Dec 26 18:22:22.306790919 (506h53m22.177873964s) [noise=true , address=145.79.227.42:4123 , piece_id=DYVDISG4ZZSGO7TPYQCFM4ZSLCFUR7F35EFWJBLOF47EIKAPVTMQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.317440578 (506h53m22.188523623s) [noise=true , address=145.79.227.48:4303 , piece_id=EKVOCHB53K4HIN2JFZGRZPCJJUVIAC6BYKXPMEKJCTTNL7YSJDVA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.324498949 (506h53m22.195581994s) [noise=true , address=145.79.227.46:4602 , piece_id=HH6MVNSJQCFI4TGAGLNCJS37LS7RB37KP5NDL4JAJJNKBIR7JTTQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.332376839 (506h53m22.203459874s) [noise=true , address=145.79.227.45:4230 , piece_id=3F5NRWI4RRVB7GJBHW3HH34G7MAJBGQ47OWNPHKNDXZYBNYE6MWA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.335008621 (506h53m22.206091666s) [noise=true , address=145.79.227.8:4330 , piece_id=VUBUBFR73QD4K47IDNI24B6T3KKOUV7VIHO4QI3JFYFOYTHZ76BA, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.337891891 (506h53m22.208974926s) [noise=true , address=145.79.227.51:4410 , piece_id=54EBW2MUNATUOVLSISA7FNI7DKUOEDSN2MNOEUTQHDDQRHBE54XQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.338036998 (506h53m22.209120033s) [noise=true , address=145.79.227.49:4339 , piece_id=EGL3XFJKGTELBWBI5DSKC7JWESOLRM2XMO6LRS4WCB6IJZWZ6CUQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.357245244 (506h53m22.228328289s) [noise=true , address=145.79.227.6:4257 , piece_id=Q4EUKU62POICD3YVHHTR722DWDNOONOBG3LHMWGTISESZYWWF47Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.491011368 (506h53m22.362094403s) [noise=true , address=77.90.31.16:4736 , piece_id=FHQYVM44UEO4RUTAHOFIQ7LO3WYXAJGW23DSSWA6JRHJZ3RU6HHQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.492463102 (506h53m22.363546137s) [noise=true , address=77.90.31.10:4520 , piece_id=SDDHFO7ZCRH4B5PLWEXHM7ZCBLX2V5GJLFLNLO3UZPTBKRW4FU7Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.543901169 (506h53m22.414984204s) [noise=true , address=45.91.116.4:4937 , piece_id=OILDTM7QBOICW4GLPOMPFTOQGXN7KVPJER3RGT7FUSFHDMQMGUBQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.555202418 (506h53m22.426285454s) [noise=true , address=45.91.116.7:4012 , piece_id=NJC5EMFER5YYISOPZU6UARD6PUPZAHMUAF3TNB4LJPHXA4QJJO5A, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584924595 (506h53m22.45600763s) [noise=true , address=77.90.31.13:4629 , piece_id=HWLERTDU6GCEN6IQLRMIXR2TWJHJUFUFAENXZSFNDP4GYMOSQOGQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: <nil> Dec 26 18:22:22.584944242 (506h53m22.456027277s) [seg_pos={0 0}, upload=444786]: Segment reached optimal threshold of 13 pieces. Dec 26 18:22:22.585279571 (506h53m22.456362616s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585291072 (506h53m22.456374107s) [noise=true , address=145.79.227.40:4049 , piece_id=A54YZBC5INMC4AIGFW45TDVGXO554WBTIXR7POY7H2SHZXV2LKPQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585390301 (506h53m22.456473366s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54922->45.91.116.11:4118: use of closed network connection Dec 26 18:22:22.585420619 (506h53m22.456503674s) [noise=true , address=45.91.116.11:4118 , piece_id=G2O63TBJ7PM2AP6QPIVIIWSMR2EBFOOWJBQTMIB2GECGQUDCGVIQ, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585629497 (506h53m22.456712552s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 26 18:22:22.585657961 (506h53m22.456741006s) [noise=true , address=45.91.116.13:4155 , piece_id=7Z5CNXM7NK4H4KBR5WXMEVYKVCBBFYJEEVAUDC7NPRBLKMM4PT6Q, seg_pos={0 0}, upload=444786]: Long tail cancellation due to optimal threshold Dec 26 18:22:22.585679984 (506h53m22.456763029s) [seg_pos={0 0}, upload=444786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:22:22.585785955 (506h53m22.45686899s) [upload=444786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:22:22.585792888 (506h53m22.456875923s) [upload=444786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:22:22.894574533 (506h53m22.765657578s) [upload=444786]: Done uploading segments... Dec 26 18:22:22.894598457 (506h53m22.765681492s) [upload=444786]: Upload finished. err: <nil> Duration: 30.003759798s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.....] duration: 6.133632138s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 6.133659181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 15.379472877s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:22:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 15.379491543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 33.876030851s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:21:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 33.876051089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....X.XXX.....XX.X.XXX.........X..X.....X...XX.........X....] duration: 5.834379679s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:21:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 5.834412691s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 43.42257527s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:21:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 43.422589438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 41.797177059s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:20:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 41.797207607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.XXX.....XX.X.XXX.........X..X.....X...XX.........X...] duration: 6.185197297s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:20:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 6.185272379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 50.81668522s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 50.816713354s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.XXX.....XX.X.XXX.........X..X.....X...XX.........X..] duration: 6.303600129s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 6.303628111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X......X.XXX.....XX.X.XXX.........X..X.....X...XX.........X.] duration: 6.273329135s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:18:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 6.273355025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 37.77361372s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:18:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 37.773638628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.........X] duration: 30.003423559s trace: https://jaeger.dev.storj.io/trace/3b66cb5bbee176a1 FAIL (2025-12-26 18:18:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: 6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A) context deadline exceeded ======LOGS====== gments... Dec 26 18:17:44.233899075 (506h48m44.104982321s) [upload=444720]: Got next segment. Inline: false Dec 26 18:17:44.234055674 (506h48m44.105138719s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:17:44.234062016 (506h48m44.105145051s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:17:44.482104382 (506h48m44.353187437s) [seg_pos={0 0}, upload=444720]: Begin upload segment... Dec 26 18:17:44.482168003 (506h48m44.353251038s) [seg_pos={0 0}, upload=444720]: Done begin upload segment. Dec 26 18:17:44.482244459 (506h48m44.353327494s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.482284294 (506h48m44.35336734s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483125356 (506h48m44.354208401s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483466666 (506h48m44.354549701s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483697005 (506h48m44.35478005s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.483845718 (506h48m44.354928753s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484134077 (506h48m44.355217112s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484298069 (506h48m44.355381104s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484400655 (506h48m44.3554837s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484691678 (506h48m44.355774713s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484819723 (506h48m44.355902768s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.484986671 (506h48m44.356069706s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485248549 (506h48m44.356331614s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.485634844 (506h48m44.356717879s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486143163 (506h48m44.357226208s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:44.486483892 (506h48m44.357566937s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Uploading piece... Dec 26 18:17:46.098958535 (506h48m45.970041571s) [upload=444720]: Next returned nil segment Dec 26 18:17:46.098973043 (506h48m45.970056078s) [upload=444720]: Waiting for error group managing segments... Dec 26 18:17:46.482233768 (506h48m46.353316813s) [noise=true , address=145.79.227.36:4517 , piece_id=GNQIP445TCJH3DRP34QUKON66OCCGA54XZ67ILL6G5WXFHBV54UQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484253554 (506h48m46.35533662s) [noise=true , address=145.79.227.22:4511 , piece_id=JEVHL5N6JFIIZJGIMMKPGGW5O4FRYU3TLL6SGA2DXYUU7IEWFSXA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.484520213 (506h48m46.355603258s) [noise=true , address=145.79.227.49:4337 , piece_id=SXX4XTTPZFHVLPQ6MT2S6KDMQBTSUDR35Q4WZMVJSTGHR5S47VFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.485024783 (506h48m46.356107838s) [noise=true , address=145.79.227.29:4942 , piece_id=UVESXQQA4WWA6BLWFQYKKXZSB72OS5S7VOWLOTKSGC64CPMVRVCA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.490941920 (506h48m46.362024955s) [noise=true , address=145.79.227.37:4566 , piece_id=2UGNFWUMBFESEFINRGNYGP5BSZ2AGBP4V5KGMYJXIWISYGFSY7XQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491343145 (506h48m46.3624262s) [noise=true , address=145.79.227.52:4446 , piece_id=JQV34TEV346A4PWGH3CDKD2GHAP6KNVUHJ3DZJD6C34G2I23EBFA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.491517597 (506h48m46.362600642s) [noise=true , address=145.79.227.11:4731 , piece_id=MFVIRTTK4HMMEVYLEQLNJI4GHPHDSKNCKES5265OOKHP35V2BZ4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.505557661 (506h48m46.376640706s) [noise=true , address=145.79.227.47:4266 , piece_id=LMYKFSRNB4UBLK2Z7FRWKCRUIP3PTXNMSZA4VIUUHOIIMVCZEHHA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.511497923 (506h48m46.382580968s) [noise=true , address=145.79.227.31:4799 , piece_id=7VDEU4P6RWTT2SL6GFZGAYJYVGMGNEUZBX5ELRGQQME2AVI4EGYA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:46.748964861 (506h48m46.620047906s) [noise=true , address=145.79.227.7:4294 , piece_id=HMLTVW3MUFCAR6BG74SVDTNSICW72VYZX4J2YNIH4OAZA6M2MYNA, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.018364295 (506h48m46.88944734s) [noise=true , address=45.91.116.14:4190 , piece_id=HTIDJEQFTYA5OEDTOSXUUV7AVYISWN5BOEP6GD7KD7J4ON3CBPFQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.520456737 (506h48m47.391539782s) [noise=true , address=45.91.116.6:4973 , piece_id=6U4LPZYNWILR72DVNPKP22VKKNOZPWMY73OQAIEGRDVY6PFNGV7A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646618581 (506h48m47.517701626s) [noise=true , address=45.91.116.5:4298 , piece_id=P7URE5INZE6MEYA7OB2EWBMNV264UFRFWHECXGTDCNQWNDEZLUIQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: <nil> Dec 26 18:17:47.646719584 (506h48m47.517802619s) [seg_pos={0 0}, upload=444720]: Segment reached optimal threshold of 13 pieces. Dec 26 18:17:47.646919054 (506h48m47.518002089s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43536->145.79.227.4:4187: use of closed network connection Dec 26 18:17:47.646927289 (506h48m47.518010334s) [noise=true , address=145.79.227.4:4187 , piece_id=Z4HX7IC2HBZWCEC73QWCU5K57MP46MIFX2YJH4Q52AYOCBIGUT2A, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.646940303 (506h48m47.518023339s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49674->145.79.227.5:4222: use of closed network connection Dec 26 18:17:47.646950263 (506h48m47.518033298s) [noise=true , address=145.79.227.5:4222 , piece_id=X4ES5FPY3MJKP2PSV6PLHT6JPBTCPHSEIISJ3VWXBWOL6FSYIWEQ, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647272486 (506h48m47.518355522s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 18:17:47.647280201 (506h48m47.518363236s) [noise=true , address=45.91.116.9:4264 , piece_id=LH2TMZXETF4QZ2PD7ZHAYZFHUKWXA4XOODPD4PINBKASGIIWEK4Q, seg_pos={0 0}, upload=444720]: Long tail cancellation due to optimal threshold Dec 26 18:17:47.647299127 (506h48m47.518382162s) [seg_pos={0 0}, upload=444720]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:17:47.647384159 (506h48m47.518467194s) [upload=444720]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:17:47.647390200 (506h48m47.518473235s) [upload=444720]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:17:47.945659407 (506h48m47.816742442s) [upload=444720]: Done uploading segments... Dec 26 18:17:47.945688893 (506h48m47.816771928s) [upload=444720]: Upload finished. err: <nil> Duration: 30.003448316s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 27.967344048s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:17:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 27.967361531s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX......X.XXX.....XX.X.XXX.........X..X.....X...XX.........] duration: 10.029333968s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:17:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 10.029361169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 40.733296134s trace: https://jaeger.dev.storj.io/trace/665ea65f6e9ccc33 FAIL (2025-12-26 18:16:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== e=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789316979 (506h12m54.660400024s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789326507 (506h12m54.660409552s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789611339 (506h12m54.660694384s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789619275 (506h12m54.66070231s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789914006 (506h12m54.660997041s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.789949443 (506h12m54.661032478s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790181566 (506h12m54.661264601s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790295632 (506h12m54.661378668s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790510872 (506h12m54.661593917s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790692980 (506h12m54.661776015s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790787379 (506h12m54.661870414s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.790992970 (506h12m54.662076005s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791120053 (506h12m54.662203098s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791275218 (506h12m54.662358253s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:54.791443709 (506h12m54.662526744s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Uploading piece... Dec 26 17:41:56.112313145 (506h12m55.98339619s) [upload=444191]: Next returned nil segment Dec 26 17:41:56.112330638 (506h12m55.983413683s) [upload=444191]: Waiting for error group managing segments... Dec 26 17:41:56.485322720 (506h12m56.356405765s) [noise=true , address=64.49.13.138:4487 , piece_id=BM3WSVWI7LEPX3L7BTSBESEPMTUDC5MDNQ22NE2XCWQNAP7HFTLQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.485637079 (506h12m56.356720124s) [noise=true , address=64.49.13.139:4469 , piece_id=IFO7ZSQJCPLTDTCKO5JNXWEIGCDIXFM6GT3EFJ2JRSMZTBIUNQ6Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.666807539 (506h12m56.537890584s) [noise=true , address=145.79.227.36:4529 , piece_id=EHB3LQUY7DHGV4QBEWW4UCRXK2PIRGCGYN37SL244S4EWY2S5BTA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.678753987 (506h12m56.549837032s) [noise=true , address=145.79.227.49:4370 , piece_id=IT5PZJZR4UD2YZ4Z4CANH57KUTRRZRMWOLB6C7RXMRUVLL6PPGQQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.687788039 (506h12m56.558871074s) [noise=true , address=145.79.227.46:4604 , piece_id=2EMBQTQZQTZLRW42SX2UBLX6O4MADVELLTWFINCNPMP6D2UU4SCQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.694318023 (506h12m56.565401058s) [noise=true , address=145.79.227.7:4326 , piece_id=DMQ5RPI46X2U7TJTS42ZHQEF32BDJRKJJO5HWOYAPNS6VWVG5B5Q, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.716581187 (506h12m56.587664232s) [noise=true , address=145.79.227.31:4803 , piece_id=JLQPVKM32D3B5HWSXWJMHN6NBL7TC7ADMMSFCHVCXC6LF6WEQ2GA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:56.972183007 (506h12m56.843266042s) [noise=true , address=45.91.116.7:4043 , piece_id=PVR6ADM2YWV6P76V6X74KQUNEZIX6HCGL3RUAHU4FWU232GEODWA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.011917999 (506h12m56.883001034s) [noise=true , address=77.90.31.21:4890 , piece_id=W4HVOCPJ6KEZGM7XSUHW5Z7ZNV5PNCNZMV6DUEIVVNAU26ZJQXTQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.054168622 (506h12m56.925251667s) [noise=true , address=45.91.116.8:4062 , piece_id=LMB5A3LN3HDS3GBLU2PYYAQBG5L7NB6G4FRWP3SPDBLT3F44OTDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.114312777 (506h12m56.985395812s) [noise=true , address=77.90.31.19:4849 , piece_id=X4LKAC4ZWHRAJEOO7DANBEJJFCHBSAQ5OBNJIOLCKDQX3ABF5RMQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.499846250 (506h12m57.370929285s) [noise=true , address=145.79.227.34:4919 , piece_id=YRWHW3BJVEGVJRSA6RJB3T7VJQSDTGKEZQKRICFIV2G3T7AN4MIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.612649066 (506h12m57.483732101s) [noise=true , address=145.79.227.40:4081 , piece_id=ZWSYVGIAE6TL53BTS5VW2WDGS5PSQGHFTPNXLLE6NKODQNXAD4HQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649413733 (506h12m57.520496768s) [noise=true , address=145.79.227.30:4789 , piece_id=RGT7T4ETW35LEWVFZIMQZJ5RIUBNGT5YRCIQ7J5T3DQGFPA2XDIQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: <nil> Dec 26 17:41:57.649440282 (506h12m57.520523328s) [seg_pos={0 0}, upload=444191]: Segment reached optimal threshold of 14 pieces. Dec 26 17:41:57.649674729 (506h12m57.520757774s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G7ty71tPwe7cZSBNK9u74DFUCGomnxDsNvAif4C3SbP7HWwfo): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649684087 (506h12m57.520767142s) [noise=true , address=77.90.31.15:4731 , piece_id=KPVZXU5IKBJ74OVZDCE6CX5W2U77QTLIG7WOFGARKQ6WQTPDISUA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649747708 (506h12m57.520830763s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu): protocol: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38500->145.79.227.41:4103: use of closed network connection Dec 26 17:41:57.649785219 (506h12m57.520868274s) [noise=true , address=145.79.227.41:4103 , piece_id=O26UJQBJKQ3APZ4EIRX5QHMIP7QVMZM6AGQB5PIDSELZDIYWYPAQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.649899828 (506h12m57.520982903s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ve7KKYDbNo8Ma7wGVBH5x9z7zbPtPAPztmuQ9r6ijPpVgh5NX): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.649908775 (506h12m57.52099182s) [noise=true , address=77.90.31.13:4649 , piece_id=WQWVOJGIFGXL6RD5LJKJFTNKO4H6TUMGYIMB2VZ7ZUOYH236LLDQ, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650074641 (506h12m57.521157696s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3): protocol: expected piece hash; context canceled; EOF Dec 26 17:41:57.650083287 (506h12m57.521166342s) [noise=true , address=77.90.31.12:4621 , piece_id=IJ4YMIQRF3CUQA7AC7KOPJ3K2OTF7J5PGRVDI5BA4NYXGKF6H5IA, seg_pos={0 0}, upload=444191]: Long tail cancellation due to optimal threshold Dec 26 17:41:57.650132821 (506h12m57.521215866s) [seg_pos={0 0}, upload=444191]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 17:41:57.650220138 (506h12m57.521303183s) [upload=444191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 17:41:57.650227171 (506h12m57.521310206s) [upload=444191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 17:41:57.958890870 (506h12m57.829973915s) [upload=444191]: Done uploading segments... Dec 26 17:41:57.958911620 (506h12m57.829994665s) [upload=444191]: Upload finished. err: <nil> Duration: 40.733322695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XX......X.XXX.....XX.X.XXX.........X..X.....X...XX........] duration: 5.836468248s trace: https://jaeger.dev.storj.io/trace/3992379534f794fb FAIL (2025-12-26 18:16:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA) context deadline exceeded ======LOGS====== Starting upload Dec 26 18:11:00.104273846 (506h41m59.975356892s) [upload=444620]: Done starting upload Dec 26 18:11:00.104464159 (506h41m59.975547194s) [upload=444620]: Uploading segments... Dec 26 18:11:00.104513854 (506h41m59.975596889s) [upload=444620]: Got next segment. Inline: false Dec 26 18:11:00.104544612 (506h41m59.975627647s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 18:11:00.104549912 (506h41m59.975632957s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 18:11:00.350198911 (506h42m0.221281966s) [seg_pos={0 0}, upload=444620]: Begin upload segment... Dec 26 18:11:00.350254707 (506h42m0.221337752s) [seg_pos={0 0}, upload=444620]: Done begin upload segment. Dec 26 18:11:00.350321845 (506h42m0.22140489s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350328608 (506h42m0.221411653s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.350959219 (506h42m0.222042264s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351321138 (506h42m0.222404173s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.351684470 (506h42m0.222767515s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352040457 (506h42m0.223123492s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352047270 (506h42m0.223130305s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352385835 (506h42m0.22346888s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352443936 (506h42m0.223526971s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352694222 (506h42m0.223777257s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.352808039 (506h42m0.223891084s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353054909 (506h42m0.224137944s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.353570130 (506h42m0.224653175s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354135146 (506h42m0.225218181s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.354603829 (506h42m0.225686874s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:00.355020522 (506h42m0.226103567s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Uploading piece... Dec 26 18:11:02.032410415 (506h42m1.90349346s) [upload=444620]: Next returned nil segment Dec 26 18:11:02.032430152 (506h42m1.903513197s) [upload=444620]: Waiting for error group managing segments... Dec 26 18:11:02.348324681 (506h42m2.219407716s) [noise=true , address=145.79.227.22:4511 , piece_id=S6ZS6SOLNLZXGX7H4CUFN565Y4AYUIZCC2CXBJS6L3CSVB4OC3XQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.348549048 (506h42m2.219632083s) [noise=true , address=145.79.227.38:4978 , piece_id=6UFEMERR2PHRITTQKOGZV3CZCYLHONY36POKRX4TJ3IG3LB4RLQQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.351071442 (506h42m2.222154487s) [noise=true , address=145.79.227.8:4330 , piece_id=PDB6ALO5ZPYSMSEQHF3NPY3VPWG7XGCXPXA22HWBFRGA5YYA3G5A, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.355344347 (506h42m2.226427392s) [noise=true , address=145.79.227.50:4374 , piece_id=TMHRSFYNG6WI6VZF6CER7JWWG4PAAMSJ22QP6RAI64CJO4GAHNGQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.359503999 (506h42m2.230587044s) [noise=true , address=145.79.227.49:4337 , piece_id=BUOWQXWRHHZCKHJPC3D6KNYPHLNNVNSYZCMLG4JICHTJZVFVB7VQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.361316750 (506h42m2.232399785s) [noise=true , address=145.79.227.5:4223 , piece_id=5SXLARFJKURKC7AIF765Z7MWPHOEDHOFMNLCHMQBZUYBOC6AVNOQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.373894200 (506h42m2.244977235s) [noise=true , address=145.79.227.47:4267 , piece_id=XRQQF5K3O3A4V5545MZA2YG2N6EVRG4KIM56A6W6IRRWISTU6WAQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.376201063 (506h42m2.247284108s) [noise=true , address=145.79.227.36:4517 , piece_id=ZL6T3RQT5XNQURM6JLOZ2YGAXNGLYGN5FYHO47IKWPPXZFCUEVBA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.380841749 (506h42m2.251924804s) [noise=true , address=145.79.227.11:4731 , piece_id=NJN4CQGBUBFZWEUDP3KBVU2AVACTXCQ2HE6YJ7W7MKC4E3D4LXIQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.388670316 (506h42m2.259753351s) [noise=true , address=145.79.227.42:4121 , piece_id=H56KQOPDWK557ZZJCWBH7YLWWL3DK24BO3654Q4EWQ6ALZ5AU2UQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.586101119 (506h42m2.457184164s) [noise=true , address=45.91.116.7:4009 , piece_id=LLOE3CT5REHRQEWDHHSGDBGJPWC35OQMVR4MEZB4FDOZ4AWGFGHA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.597900427 (506h42m2.468983462s) [noise=true , address=45.91.116.10:4083 , piece_id=OBCQWLK3PCTYKVBP6DPVWXVQPJZVDVHMXS7333U5V5ROKQ2KQEEQ, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711289730 (506h42m2.582372765s) [noise=true , address=45.91.116.6:4974 , piece_id=DLDNZ7WRHTHESIBXZBLTGAQNLJRAJBDKPZHSCQZW6GYUNLYIONDA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: <nil> Dec 26 18:11:02.711310149 (506h42m2.582393184s) [seg_pos={0 0}, upload=444620]: Segment reached optimal threshold of 13 pieces. Dec 26 18:11:02.711579893 (506h42m2.582662938s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:54212->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 18:11:02.711589541 (506h42m2.582672576s) [noise=true , address=45.91.116.11:4119 , piece_id=GA4DRGZTVG4TKE6MPO2YWCWERELOSSF2QZYG5L5MG7TEG5C6TFVA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711667138 (506h42m2.582750184s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 26 18:11:02.711691024 (506h42m2.582774069s) [noise=true , address=45.91.116.8:4047 , piece_id=K5Q2OWU2L6QI5Y77KA7OS5UYOGDYUZ6B6XWURJMSUQXCEUAFC5UA, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711887848 (506h42m2.582970913s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38572->45.91.116.5:4297: use of closed network connection Dec 26 18:11:02.711896776 (506h42m2.582979821s) [noise=true , address=45.91.116.5:4297 , piece_id=APSDM2AGSPHTXZ36DROJOVMPFVPCHUHY5UBTILU32LERBQQOWK2Q, seg_pos={0 0}, upload=444620]: Long tail cancellation due to optimal threshold Dec 26 18:11:02.711919389 (506h42m2.583002434s) [seg_pos={0 0}, upload=444620]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 18:11:02.712023507 (506h42m2.583106542s) [upload=444620]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 18:11:02.712031572 (506h42m2.583114607s) [upload=444620]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 18:11:03.015981797 (506h42m2.887064842s) [upload=444620]: Done uploading segments... Dec 26 18:11:03.016006694 (506h42m2.887089739s) [upload=444620]: Upload finished. err: <nil> Duration: 5.836499206s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)