Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..........................X.......................X........] duration: 5.806841221s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.806862902s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..........................X.......................X.......] duration: 5.942259619s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:31:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.942292381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X..........................X.......................X......] duration: 5.946696087s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:30:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.946726364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X..........................X.......................X.....] duration: 5.798287854s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:30:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.798376864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..........................X.......................X....] duration: 5.963464568s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:29:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.963494896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X..........................X.......................X...] duration: 5.974084822s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:28:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.974112044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X..........................X.......................X..] duration: 21.47074092s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 21.470757803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X..........................X.......................X.] duration: 6.06485971s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:27:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 6.064890328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X..........................X.......................X] duration: 30.003400864s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:26:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 30.003422185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..........................X.......................] duration: 6.36324811s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:25:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 6.363284268s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..........................X......................] duration: 6.242271749s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:24:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 6.242306396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..........................X.....................] duration: 5.829408354s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:24:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.829437599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..........................X....................] duration: 28.132088724s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:23:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 28.132110455s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.587992099s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:23:07 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.588027207s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X............X..........................X...................] duration: 5.91775762s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:22:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.917785936s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.289297735s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:22:26 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.289325286s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.583342637s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:22:08 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.583373956s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.319320093s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:21:41 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.319347576s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X............X..........................X..................] duration: 5.661907376s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:21:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.661935018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X............X..........................X.................] duration: 5.962686408s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:21:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.962713892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X............X..........................X................] duration: 5.725939652s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:20:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.725971513s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.495617742s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:20:40 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.495639404s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X............X..........................X...............] duration: 5.843921195s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:20:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.843974298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X..........................X..............] duration: 5.898158374s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:19:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.89817253s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.825584405s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:19:33 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.825611526s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X............X..........................X.............] duration: 5.910194416s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:19:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.910213313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X............X..........................X............] duration: 29.666891202s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:18:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 29.666923863s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.235317449s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:18:32 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.235344161s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.155119791s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:18:03 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.155147534s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X............X..........................X...........] duration: 5.637435537s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:17:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.637465573s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.720112708s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:17:15 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.720144158s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X............X..........................X..........] duration: 6.558363875s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:17:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 6.558392249s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.372364251s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:16:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.372389832s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.202035005s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:16:14 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.202064701s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.296616944s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:15:59 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 6.296645108s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X............X..........................X.........] duration: 5.575883463s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:15:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.57591282s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.435135299s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:15:50 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.43515107s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.02433304s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:15:17 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.024361123s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.374265052s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:15:05 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.374299438s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X............X..........................X........] duration: 5.654745848s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:14:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.654774683s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.644519099s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:14:32 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.644564475s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X............X..........................X.......] duration: 6.795012964s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:14:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 6.795035997s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.359182753s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:14:11 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.359216538s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.414173176s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:13:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.414203463s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.186769214s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:13:47 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.186799992s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X............X..........................X......] duration: 5.709629977s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:13:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.709661017s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.43300409s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:13:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.43304629s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.18066s trace: https://jaeger.dev.storj.io/trace/3a2fe70d61d2edae FAIL (2025-12-23 16:13:14 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.180691872s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.391357487s trace: https://jaeger.dev.storj.io/trace/291af7ecf87bc602 FAIL (2025-12-23 16:12:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.391384698s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X............X..........................X.....] duration: 5.696768251s trace: https://jaeger.dev.storj.io/trace/7202769704e59d86 FAIL (2025-12-23 16:12:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA) context deadline exceeded; piecestore: (Node ID: 12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, Piece ID: XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ) context deadline exceeded ======LOGS====== Dec 23 16:08:35.937446291 (432h39m35.808529336s) [upload=379809]: Starting upload Dec 23 16:08:35.937507067 (432h39m35.808590102s) [upload=379809]: Done starting upload Dec 23 16:08:35.937557633 (432h39m35.808640668s) [upload=379809]: Uploading segments... Dec 23 16:08:35.937699183 (432h39m35.808782228s) [upload=379809]: Got next segment. Inline: false Dec 23 16:08:35.937734380 (432h39m35.808817425s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:08:35.937739089 (432h39m35.808822124s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:08:36.318856960 (432h39m36.189940005s) [seg_pos={0 0}, upload=379809]: Begin upload segment... Dec 23 16:08:36.318915000 (432h39m36.189998045s) [seg_pos={0 0}, upload=379809]: Done begin upload segment. Dec 23 16:08:36.318949897 (432h39m36.190032942s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319112276 (432h39m36.190195311s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319603752 (432h39m36.190686797s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319612118 (432h39m36.190695153s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319889325 (432h39m36.190972361s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.319974107 (432h39m36.191057153s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320163538 (432h39m36.191246573s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320507142 (432h39m36.191590187s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320634084 (432h39m36.191717129s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.320869251 (432h39m36.191952286s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321023215 (432h39m36.19210625s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321222264 (432h39m36.192305309s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321364445 (432h39m36.19244748s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321561800 (432h39m36.192644835s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.321607087 (432h39m36.192690122s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:36.322025434 (432h39m36.193108479s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Uploading piece... Dec 23 16:08:38.223082161 (432h39m38.094165196s) [upload=379809]: Next returned nil segment Dec 23 16:08:38.223098812 (432h39m38.094181857s) [upload=379809]: Waiting for error group managing segments... Dec 23 16:08:38.604478490 (432h39m38.475561545s) [noise=true , address=45.91.116.7:4011 , piece_id=YXSHBDTGC2V2ATQD4LBAABU6U24O5IGM6XSQQZNB7IKPKQAIN7UQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:38.979335098 (432h39m38.850418133s) [noise=true , address=77.90.31.13:4627 , piece_id=IO37A3VMCCO5P2FA72DGZLE7D23F7FCZ3BSM7J7ZQJ7BCFCL3RGQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.265662090 (432h39m39.136745135s) [noise=true , address=45.91.116.9:4263 , piece_id=3KB7FYHHQ3UXJPQ64LHSGB4E5HQ7QTAB4GUEAOU656VBWQXM7UJA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.541862740 (432h39m39.412945785s) [noise=true , address=145.79.227.33:4869 , piece_id=FG36BWG2GHRQJ7QFJBGUAHG4FA5P5UXTUMSPCY72ZESBU5Q6ZMCQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.542653837 (432h39m39.413736872s) [noise=true , address=145.79.227.35:4481 , piece_id=6FDOTHVU2VQX5R55VPY5H7LVN7I2KIML2GPFQFGR5SMI7BHGHZZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565079800 (432h39m39.436162845s) [noise=true , address=145.79.227.45:4229 , piece_id=MEIJ42XD57MBUNVNAKINYE64VM3P7VXPOF26422KXXHSOZGSOY2Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.565748202 (432h39m39.436831237s) [noise=true , address=145.79.227.38:4977 , piece_id=3PCLZPCQSNOVSVIBCLBUKAFC7PLMDLHYFWFF4WXYRL5UGMZ6KWYQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.570202684 (432h39m39.441285729s) [noise=true , address=145.79.227.8:4329 , piece_id=2HVISDP3PKX3M6USGJBSGMOO7DVYEUN3UUBCTJBFQSPEGNF4XH6Q, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.592258612 (432h39m39.463341647s) [noise=true , address=145.79.227.2:4113 , piece_id=UU2SNYCRWKN2VACPNI3J62XLHLLO67PBKFHE3DAQI6SZSKJWAAZQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.599779423 (432h39m39.470862458s) [noise=true , address=145.79.227.29:4943 , piece_id=4XQZFHBXNMHSMHFNCE5LSWSMPRA5BUH3H5XIUAZ2TAZFBLIFKIOA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.615289076 (432h39m39.486372111s) [noise=true , address=145.79.227.46:4602 , piece_id=7UDCA5HIL5UKH3S3C2A5POWSTSGY6EOEBGNIPZ3YYWXF47TY7GTA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.620258108 (432h39m39.491341143s) [noise=true , address=145.79.227.43:4159 , piece_id=XUMIC7WMRB2HGTRWGCX46KL66XYC2PG2LD4PIACCWS3FVBU6H6YQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625487645 (432h39m39.49657068s) [noise=true , address=145.79.227.17:4970 , piece_id=LGADTXSTOIAAU2LXBZUXS67ILFOWFLNWSBZTUHCFKOT5GC7UIETQ, seg_pos={0 0}, upload=379809]: Done uploading piece. err: <nil> Dec 23 16:08:39.625514597 (432h39m39.496597642s) [seg_pos={0 0}, upload=379809]: Segment reached optimal threshold of 13 pieces. Dec 23 16:08:39.625938213 (432h39m39.497021288s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.625949796 (432h39m39.497032831s) [noise=true , address=145.79.227.4:4187 , piece_id=2XRW3VRGYU5EDIFFYGQVVYKISJRFR2OQOT6AXOH2R355IKOGMHEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626197027 (432h39m39.497280062s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626212355 (432h39m39.49729539s) [noise=true , address=145.79.227.39:4015 , piece_id=EM3JPKCL7JABXLFXVDU7F42YYCUXXLDN6RVIWYYWUROQ3VYEDWZA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626329960 (432h39m39.497412995s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: expected piece hash; context canceled; EOF Dec 23 16:08:39.626339849 (432h39m39.497422894s) [noise=true , address=145.79.227.48:4303 , piece_id=EP2EJAWI2D4CLOISZNEUDK2RMJWJ66EHJ65QZ733SZY2X5LCLSEA, seg_pos={0 0}, upload=379809]: Long tail cancellation due to optimal threshold Dec 23 16:08:39.626366710 (432h39m39.497449745s) [seg_pos={0 0}, upload=379809]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:08:39.626450719 (432h39m39.497533765s) [upload=379809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:08:39.626459746 (432h39m39.497542781s) [upload=379809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:08:40.064332307 (432h39m39.935415352s) [upload=379809]: Done uploading segments... Dec 23 16:08:40.064355500 (432h39m39.935438535s) [upload=379809]: Upload finished. err: <nil> Duration: 5.69680013s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)