Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X....................X.........X.................X.] duration: 7.414939881s trace: https://jaeger.dev.storj.io/trace/66c7fe5cfcfeb881 FAIL (2025-12-31 23:02:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: I4SBZV32XIFP4QNIT5AUG76MHA25SIZVIRK5TUQ55CTLA6GV5SCQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: XTZNH4CZIJUVO6UKRE3SR47H7JU7BZVJJVTBL2XHI6LCVE5PRFVA) context deadline exceeded ======LOGS====== 0 (631h32m24.664502985s) [upload=553399]: Done starting upload Dec 31 23:01:24.793605152 (631h32m24.664688207s) [upload=553399]: Uploading segments... Dec 31 23:01:24.793717106 (631h32m24.664800141s) [upload=553399]: Got next segment. Inline: false Dec 31 23:01:24.793761981 (631h32m24.664845016s) [upload=553399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 23:01:24.793766540 (631h32m24.664849585s) [upload=553399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 23:01:25.037439253 (631h32m24.908522288s) [seg_pos={0 0}, upload=553399]: Begin upload segment... Dec 31 23:01:25.037480822 (631h32m24.908563867s) [seg_pos={0 0}, upload=553399]: Done begin upload segment. Dec 31 23:01:25.037505037 (631h32m24.908588072s) [noise=true , address=77.90.31.21:4877 , piece_id=LKGDDH57BSOTQC64YDLE4WRCV2AE6EOFM4GHWT7M47IPVPKWYANA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.037557658 (631h32m24.908640703s) [noise=true , address=145.79.227.13:4802 , piece_id=B6NGTPWXMZ7G5Q5YD7R343FRLZAT6I76O2AW3VA5YOWPK4DVA2AQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038277659 (631h32m24.909360694s) [noise=true , address=145.79.227.11:4729 , piece_id=DFHIXBY3AILIMKG4O6JVOCDV3YZGG77F4EJXNMSIIEJQV2S5L5NQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038352071 (631h32m24.909435106s) [noise=true , address=45.91.116.10:4083 , piece_id=I4SBZV32XIFP4QNIT5AUG76MHA25SIZVIRK5TUQ55CTLA6GV5SCQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038646451 (631h32m24.909729496s) [noise=true , address=45.91.116.6:4976 , piece_id=YLP5MJETZUGKCFAOVWCOYAKZZ6HV2JH246X4Y7P3AAPTZJNJIIRQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038843416 (631h32m24.909926451s) [noise=true , address=45.91.116.9:4264 , piece_id=CQBJBOE5AGT5B3ZVTHHLF3C46ZSRX4MOT4JPHNIFKRBRCZGDHPOQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039013450 (631h32m24.910096485s) [noise=true , address=45.91.116.8:4045 , piece_id=CUTA3TTWI3Z5NN7JO2IECQDJGMHGDZIFTXXS3HTMW6ILZJVZYBBA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039189886 (631h32m24.910272921s) [noise=true , address=45.91.116.12:4225 , piece_id=XTZNH4CZIJUVO6UKRE3SR47H7JU7BZVJJVTBL2XHI6LCVE5PRFVA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039318861 (631h32m24.910401896s) [noise=true , address=145.79.227.9:4365 , piece_id=OEZBTNMR5AMUXTRGFBVDQQ2MKZ2PVSFZUWTKUY3BWQ6IWTU3CR3Q, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039624032 (631h32m24.910707067s) [noise=true , address=77.90.31.15:4697 , piece_id=BYFHFULMZS64HLHQ24NGJSDFZO7CRP7KCYML6A3F5GJBK3EAGD7Q, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039981192 (631h32m24.911064227s) [noise=true , address=145.79.227.10:4402 , piece_id=EXFWINONFOHNE3J7SDU67XHNAVSG7HPVUVSTRVKKHXGXMGOXWIKA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.040329625 (631h32m24.91141266s) [noise=true , address=145.79.227.22:4509 , piece_id=6CR22VDREB72EBUXZUWB26TFR5NQGICAMIOT7L2IZYQAJJHEPLPA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.040600461 (631h32m24.911683496s) [noise=true , address=77.90.31.10:4517 , piece_id=ZXYPGWWMRDZGHZQTPMPNLXXT7D7MOQICWK5LOWLSR6BRGCOYDDFA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.040854004 (631h32m24.911937039s) [noise=true , address=45.91.116.5:4297 , piece_id=MUHRCZ2BVLCTKEV2EFVPF67WXHCG2PPPREXTWB5BKLXB2I4CNX6Q, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.041558665 (631h32m24.9126417s) [noise=true , address=145.79.227.16:4910 , piece_id=W7UAAURJMBFXSUGHUQZSSM5VBCPAYWRNIBPVD3NOSUBV5T4PGNCQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.042663660 (631h32m24.913746695s) [noise=true , address=77.90.31.17:4773 , piece_id=HSV3YX4YGFS7LM7NCT4YJCIQAY4JXTPQFQSRIFYLHYKMC5VOT6DQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:26.708268221 (631h32m26.579351266s) [upload=553399]: Next returned nil segment Dec 31 23:01:26.708295033 (631h32m26.579378108s) [upload=553399]: Waiting for error group managing segments... Dec 31 23:01:27.014363591 (631h32m26.885446636s) [noise=true , address=145.79.227.9:4365 , piece_id=OEZBTNMR5AMUXTRGFBVDQQ2MKZ2PVSFZUWTKUY3BWQ6IWTU3CR3Q, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.023494297 (631h32m26.894577332s) [noise=true , address=145.79.227.10:4402 , piece_id=EXFWINONFOHNE3J7SDU67XHNAVSG7HPVUVSTRVKKHXGXMGOXWIKA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.032293280 (631h32m26.903376315s) [noise=true , address=145.79.227.22:4509 , piece_id=6CR22VDREB72EBUXZUWB26TFR5NQGICAMIOT7L2IZYQAJJHEPLPA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.035464869 (631h32m26.906547914s) [noise=true , address=145.79.227.13:4802 , piece_id=B6NGTPWXMZ7G5Q5YD7R343FRLZAT6I76O2AW3VA5YOWPK4DVA2AQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.051286274 (631h32m26.922369319s) [noise=true , address=145.79.227.11:4729 , piece_id=DFHIXBY3AILIMKG4O6JVOCDV3YZGG77F4EJXNMSIIEJQV2S5L5NQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.053406181 (631h32m26.924489216s) [noise=true , address=145.79.227.16:4910 , piece_id=W7UAAURJMBFXSUGHUQZSSM5VBCPAYWRNIBPVD3NOSUBV5T4PGNCQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.240957724 (631h32m27.112040759s) [noise=true , address=77.90.31.21:4877 , piece_id=LKGDDH57BSOTQC64YDLE4WRCV2AE6EOFM4GHWT7M47IPVPKWYANA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.271727432 (631h32m27.142810467s) [noise=true , address=77.90.31.17:4773 , piece_id=HSV3YX4YGFS7LM7NCT4YJCIQAY4JXTPQFQSRIFYLHYKMC5VOT6DQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.336023396 (631h32m27.207106431s) [noise=true , address=45.91.116.5:4297 , piece_id=MUHRCZ2BVLCTKEV2EFVPF67WXHCG2PPPREXTWB5BKLXB2I4CNX6Q, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.354675601 (631h32m27.225758646s) [noise=true , address=45.91.116.12:4225 , piece_id=XTZNH4CZIJUVO6UKRE3SR47H7JU7BZVJJVTBL2XHI6LCVE5PRFVA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.368032705 (631h32m27.23911574s) [noise=true , address=45.91.116.6:4976 , piece_id=YLP5MJETZUGKCFAOVWCOYAKZZ6HV2JH246X4Y7P3AAPTZJNJIIRQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.374895061 (631h32m27.245978106s) [noise=true , address=45.91.116.9:4264 , piece_id=CQBJBOE5AGT5B3ZVTHHLF3C46ZSRX4MOT4JPHNIFKRBRCZGDHPOQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.387568342 (631h32m27.258651377s) [noise=true , address=45.91.116.10:4083 , piece_id=I4SBZV32XIFP4QNIT5AUG76MHA25SIZVIRK5TUQ55CTLA6GV5SCQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.387586165 (631h32m27.2586692s) [seg_pos={0 0}, upload=553399]: Segment reached optimal threshold of 13 pieces. Dec 31 23:01:27.387751200 (631h32m27.258834235s) [noise=true , address=77.90.31.10:4517 , piece_id=ZXYPGWWMRDZGHZQTPMPNLXXT7D7MOQICWK5LOWLSR6BRGCOYDDFA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:36614->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:36614->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36614->77.90.31.10:4517: use of closed network connection Dec 31 23:01:27.387760067 (631h32m27.258843102s) [noise=true , address=77.90.31.10:4517 , piece_id=ZXYPGWWMRDZGHZQTPMPNLXXT7D7MOQICWK5LOWLSR6BRGCOYDDFA, seg_pos={0 0}, upload=553399]: Long tail cancellation due to optimal threshold Dec 31 23:01:27.388055970 (631h32m27.259139006s) [noise=true , address=77.90.31.15:4697 , piece_id=BYFHFULMZS64HLHQ24NGJSDFZO7CRP7KCYML6A3F5GJBK3EAGD7Q, seg_pos={0 0}, upload=553399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: read tcp 188.166.225.170:38876->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 23:01:27.388062372 (631h32m27.259145408s) [noise=true , address=45.91.116.8:4045 , piece_id=CUTA3TTWI3Z5NN7JO2IECQDJGMHGDZIFTXXS3HTMW6ILZJVZYBBA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 23:01:27.388084204 (631h32m27.259167249s) [noise=true , address=45.91.116.8:4045 , piece_id=CUTA3TTWI3Z5NN7JO2IECQDJGMHGDZIFTXXS3HTMW6ILZJVZYBBA, seg_pos={0 0}, upload=553399]: Long tail cancellation due to optimal threshold Dec 31 23:01:27.388107448 (631h32m27.259190593s) [noise=true , address=77.90.31.15:4697 , piece_id=BYFHFULMZS64HLHQ24NGJSDFZO7CRP7KCYML6A3F5GJBK3EAGD7Q, seg_pos={0 0}, upload=553399]: Long tail cancellation due to optimal threshold Dec 31 23:01:27.388127226 (631h32m27.259210261s) [seg_pos={0 0}, upload=553399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 23:01:27.388199854 (631h32m27.259282889s) [upload=553399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 23:01:27.388204603 (631h32m27.259287638s) [upload=553399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 23:01:27.683423553 (631h32m27.554506609s) [upload=553399]: Done uploading segments... Dec 31 23:01:27.683447910 (631h32m27.554530945s) [upload=553399]: Upload finished. err: <nil> Duration: 7.414966913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X....................X.........X.................X] duration: 30.003085138s trace: https://jaeger.dev.storj.io/trace/66c7fe5cfcfeb881 FAIL (2025-12-31 23:01:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: I4SBZV32XIFP4QNIT5AUG76MHA25SIZVIRK5TUQ55CTLA6GV5SCQ) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: XTZNH4CZIJUVO6UKRE3SR47H7JU7BZVJJVTBL2XHI6LCVE5PRFVA) context deadline exceeded ======LOGS====== 0 (631h32m24.664502985s) [upload=553399]: Done starting upload Dec 31 23:01:24.793605152 (631h32m24.664688207s) [upload=553399]: Uploading segments... Dec 31 23:01:24.793717106 (631h32m24.664800141s) [upload=553399]: Got next segment. Inline: false Dec 31 23:01:24.793761981 (631h32m24.664845016s) [upload=553399]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 23:01:24.793766540 (631h32m24.664849585s) [upload=553399]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 23:01:25.037439253 (631h32m24.908522288s) [seg_pos={0 0}, upload=553399]: Begin upload segment... Dec 31 23:01:25.037480822 (631h32m24.908563867s) [seg_pos={0 0}, upload=553399]: Done begin upload segment. Dec 31 23:01:25.037505037 (631h32m24.908588072s) [noise=true , address=77.90.31.21:4877 , piece_id=LKGDDH57BSOTQC64YDLE4WRCV2AE6EOFM4GHWT7M47IPVPKWYANA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.037557658 (631h32m24.908640703s) [noise=true , address=145.79.227.13:4802 , piece_id=B6NGTPWXMZ7G5Q5YD7R343FRLZAT6I76O2AW3VA5YOWPK4DVA2AQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038277659 (631h32m24.909360694s) [noise=true , address=145.79.227.11:4729 , piece_id=DFHIXBY3AILIMKG4O6JVOCDV3YZGG77F4EJXNMSIIEJQV2S5L5NQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038352071 (631h32m24.909435106s) [noise=true , address=45.91.116.10:4083 , piece_id=I4SBZV32XIFP4QNIT5AUG76MHA25SIZVIRK5TUQ55CTLA6GV5SCQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038646451 (631h32m24.909729496s) [noise=true , address=45.91.116.6:4976 , piece_id=YLP5MJETZUGKCFAOVWCOYAKZZ6HV2JH246X4Y7P3AAPTZJNJIIRQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.038843416 (631h32m24.909926451s) [noise=true , address=45.91.116.9:4264 , piece_id=CQBJBOE5AGT5B3ZVTHHLF3C46ZSRX4MOT4JPHNIFKRBRCZGDHPOQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039013450 (631h32m24.910096485s) [noise=true , address=45.91.116.8:4045 , piece_id=CUTA3TTWI3Z5NN7JO2IECQDJGMHGDZIFTXXS3HTMW6ILZJVZYBBA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039189886 (631h32m24.910272921s) [noise=true , address=45.91.116.12:4225 , piece_id=XTZNH4CZIJUVO6UKRE3SR47H7JU7BZVJJVTBL2XHI6LCVE5PRFVA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039318861 (631h32m24.910401896s) [noise=true , address=145.79.227.9:4365 , piece_id=OEZBTNMR5AMUXTRGFBVDQQ2MKZ2PVSFZUWTKUY3BWQ6IWTU3CR3Q, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039624032 (631h32m24.910707067s) [noise=true , address=77.90.31.15:4697 , piece_id=BYFHFULMZS64HLHQ24NGJSDFZO7CRP7KCYML6A3F5GJBK3EAGD7Q, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.039981192 (631h32m24.911064227s) [noise=true , address=145.79.227.10:4402 , piece_id=EXFWINONFOHNE3J7SDU67XHNAVSG7HPVUVSTRVKKHXGXMGOXWIKA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.040329625 (631h32m24.91141266s) [noise=true , address=145.79.227.22:4509 , piece_id=6CR22VDREB72EBUXZUWB26TFR5NQGICAMIOT7L2IZYQAJJHEPLPA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.040600461 (631h32m24.911683496s) [noise=true , address=77.90.31.10:4517 , piece_id=ZXYPGWWMRDZGHZQTPMPNLXXT7D7MOQICWK5LOWLSR6BRGCOYDDFA, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.040854004 (631h32m24.911937039s) [noise=true , address=45.91.116.5:4297 , piece_id=MUHRCZ2BVLCTKEV2EFVPF67WXHCG2PPPREXTWB5BKLXB2I4CNX6Q, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.041558665 (631h32m24.9126417s) [noise=true , address=145.79.227.16:4910 , piece_id=W7UAAURJMBFXSUGHUQZSSM5VBCPAYWRNIBPVD3NOSUBV5T4PGNCQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:25.042663660 (631h32m24.913746695s) [noise=true , address=77.90.31.17:4773 , piece_id=HSV3YX4YGFS7LM7NCT4YJCIQAY4JXTPQFQSRIFYLHYKMC5VOT6DQ, seg_pos={0 0}, upload=553399]: Uploading piece... Dec 31 23:01:26.708268221 (631h32m26.579351266s) [upload=553399]: Next returned nil segment Dec 31 23:01:26.708295033 (631h32m26.579378108s) [upload=553399]: Waiting for error group managing segments... Dec 31 23:01:27.014363591 (631h32m26.885446636s) [noise=true , address=145.79.227.9:4365 , piece_id=OEZBTNMR5AMUXTRGFBVDQQ2MKZ2PVSFZUWTKUY3BWQ6IWTU3CR3Q, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.023494297 (631h32m26.894577332s) [noise=true , address=145.79.227.10:4402 , piece_id=EXFWINONFOHNE3J7SDU67XHNAVSG7HPVUVSTRVKKHXGXMGOXWIKA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.032293280 (631h32m26.903376315s) [noise=true , address=145.79.227.22:4509 , piece_id=6CR22VDREB72EBUXZUWB26TFR5NQGICAMIOT7L2IZYQAJJHEPLPA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.035464869 (631h32m26.906547914s) [noise=true , address=145.79.227.13:4802 , piece_id=B6NGTPWXMZ7G5Q5YD7R343FRLZAT6I76O2AW3VA5YOWPK4DVA2AQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.051286274 (631h32m26.922369319s) [noise=true , address=145.79.227.11:4729 , piece_id=DFHIXBY3AILIMKG4O6JVOCDV3YZGG77F4EJXNMSIIEJQV2S5L5NQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.053406181 (631h32m26.924489216s) [noise=true , address=145.79.227.16:4910 , piece_id=W7UAAURJMBFXSUGHUQZSSM5VBCPAYWRNIBPVD3NOSUBV5T4PGNCQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.240957724 (631h32m27.112040759s) [noise=true , address=77.90.31.21:4877 , piece_id=LKGDDH57BSOTQC64YDLE4WRCV2AE6EOFM4GHWT7M47IPVPKWYANA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.271727432 (631h32m27.142810467s) [noise=true , address=77.90.31.17:4773 , piece_id=HSV3YX4YGFS7LM7NCT4YJCIQAY4JXTPQFQSRIFYLHYKMC5VOT6DQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.336023396 (631h32m27.207106431s) [noise=true , address=45.91.116.5:4297 , piece_id=MUHRCZ2BVLCTKEV2EFVPF67WXHCG2PPPREXTWB5BKLXB2I4CNX6Q, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.354675601 (631h32m27.225758646s) [noise=true , address=45.91.116.12:4225 , piece_id=XTZNH4CZIJUVO6UKRE3SR47H7JU7BZVJJVTBL2XHI6LCVE5PRFVA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.368032705 (631h32m27.23911574s) [noise=true , address=45.91.116.6:4976 , piece_id=YLP5MJETZUGKCFAOVWCOYAKZZ6HV2JH246X4Y7P3AAPTZJNJIIRQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.374895061 (631h32m27.245978106s) [noise=true , address=45.91.116.9:4264 , piece_id=CQBJBOE5AGT5B3ZVTHHLF3C46ZSRX4MOT4JPHNIFKRBRCZGDHPOQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.387568342 (631h32m27.258651377s) [noise=true , address=45.91.116.10:4083 , piece_id=I4SBZV32XIFP4QNIT5AUG76MHA25SIZVIRK5TUQ55CTLA6GV5SCQ, seg_pos={0 0}, upload=553399]: Done uploading piece. err: <nil> Dec 31 23:01:27.387586165 (631h32m27.2586692s) [seg_pos={0 0}, upload=553399]: Segment reached optimal threshold of 13 pieces. Dec 31 23:01:27.387751200 (631h32m27.258834235s) [noise=true , address=77.90.31.10:4517 , piece_id=ZXYPGWWMRDZGHZQTPMPNLXXT7D7MOQICWK5LOWLSR6BRGCOYDDFA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:36614->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:36614->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36614->77.90.31.10:4517: use of closed network connection Dec 31 23:01:27.387760067 (631h32m27.258843102s) [noise=true , address=77.90.31.10:4517 , piece_id=ZXYPGWWMRDZGHZQTPMPNLXXT7D7MOQICWK5LOWLSR6BRGCOYDDFA, seg_pos={0 0}, upload=553399]: Long tail cancellation due to optimal threshold Dec 31 23:01:27.388055970 (631h32m27.259139006s) [noise=true , address=77.90.31.15:4697 , piece_id=BYFHFULMZS64HLHQ24NGJSDFZO7CRP7KCYML6A3F5GJBK3EAGD7Q, seg_pos={0 0}, upload=553399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: read tcp 188.166.225.170:38876->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 31 23:01:27.388062372 (631h32m27.259145408s) [noise=true , address=45.91.116.8:4045 , piece_id=CUTA3TTWI3Z5NN7JO2IECQDJGMHGDZIFTXXS3HTMW6ILZJVZYBBA, seg_pos={0 0}, upload=553399]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 23:01:27.388084204 (631h32m27.259167249s) [noise=true , address=45.91.116.8:4045 , piece_id=CUTA3TTWI3Z5NN7JO2IECQDJGMHGDZIFTXXS3HTMW6ILZJVZYBBA, seg_pos={0 0}, upload=553399]: Long tail cancellation due to optimal threshold Dec 31 23:01:27.388107448 (631h32m27.259190593s) [noise=true , address=77.90.31.15:4697 , piece_id=BYFHFULMZS64HLHQ24NGJSDFZO7CRP7KCYML6A3F5GJBK3EAGD7Q, seg_pos={0 0}, upload=553399]: Long tail cancellation due to optimal threshold Dec 31 23:01:27.388127226 (631h32m27.259210261s) [seg_pos={0 0}, upload=553399]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 23:01:27.388199854 (631h32m27.259282889s) [upload=553399]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 23:01:27.388204603 (631h32m27.259287638s) [upload=553399]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 23:01:27.683423553 (631h32m27.554506609s) [upload=553399]: Done uploading segments... Dec 31 23:01:27.683447910 (631h32m27.554530945s) [upload=553399]: Upload finished. err: <nil> Duration: 30.003107241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....................X.........X.................] duration: 6.263604127s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 23:01:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 6.263638963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....................X.........X................] duration: 8.020514782s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 23:00:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 8.020618801s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....................X.........X...............] duration: 14.452109825s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:59:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 14.45213866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X....................X.........X..............] duration: 6.110820521s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:58:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 6.110844668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X....................X.........X.............] duration: 5.92878768s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:57:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 5.928809011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X....................X.........X............] duration: 21.894544515s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 21.894598929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X....................X.........X...........] duration: 26.768369951s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:56:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 26.768456345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X....................X.........X..........] duration: 6.127226837s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:55:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 6.127252886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..................X....................X.........X.........] duration: 8.850323351s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 8.850354151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX..................X....................X.........X........] duration: 27.89528336s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:54:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 27.895310482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX..................X....................X.........X.......] duration: 6.177571614s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:53:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 6.177599146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX..................X....................X.........X......] duration: 6.373571196s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:52:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 6.373602585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX..................X....................X.........X.....] duration: 5.968384696s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:51:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 5.96842419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX..................X....................X.........X....] duration: 5.963596392s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:51:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 5.963662648s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX..................X....................X.........X...] duration: 6.049709651s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:50:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 6.049796507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......XX..................X....................X.........X..] duration: 7.879090361s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:49:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 7.879113115s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......XX..................X....................X.........X.] duration: 5.968835948s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:48:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 5.968853543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX..................X....................X.........X] duration: 30.003197254s trace: https://jaeger.dev.storj.io/trace/4a23f1c6d1fda99f FAIL (2025-12-31 22:48:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ) context deadline exceeded ======LOGS====== Dec 31 22:48:03.139454829 (631h19m3.010537874s) [upload=553209]: Starting upload Dec 31 22:48:03.139577823 (631h19m3.010660858s) [upload=553209]: Done starting upload Dec 31 22:48:03.139688794 (631h19m3.010771839s) [upload=553209]: Uploading segments... Dec 31 22:48:03.139946886 (631h19m3.011029921s) [upload=553209]: Got next segment. Inline: false Dec 31 22:48:03.140003694 (631h19m3.011086739s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:48:03.140087314 (631h19m3.011170359s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:48:03.387293097 (631h19m3.258376142s) [seg_pos={0 0}, upload=553209]: Begin upload segment... Dec 31 22:48:03.387340927 (631h19m3.258423962s) [seg_pos={0 0}, upload=553209]: Done begin upload segment. Dec 31 22:48:03.387370183 (631h19m3.258453228s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387425238 (631h19m3.258508344s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.387976177 (631h19m3.259059222s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388498561 (631h19m3.259581607s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.388855170 (631h19m3.259938215s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389143620 (631h19m3.260226655s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389496773 (631h19m3.260579808s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.389779451 (631h19m3.260862496s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390147741 (631h19m3.261230776s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390249545 (631h19m3.26133258s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390559155 (631h19m3.2616422s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390686057 (631h19m3.261769102s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.390901006 (631h19m3.261984051s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391216697 (631h19m3.262299742s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.391619885 (631h19m3.26270293s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:03.395883874 (631h19m3.266966909s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Uploading piece... Dec 31 22:48:04.996785600 (631h19m4.867868635s) [upload=553209]: Next returned nil segment Dec 31 22:48:04.996797963 (631h19m4.867880998s) [upload=553209]: Waiting for error group managing segments... Dec 31 22:48:05.379425832 (631h19m5.250508867s) [noise=true , address=145.79.227.19:4077 , piece_id=IPD3SI3K4RAU5NNSYBIDV6YQELZZSHVBGJLQBWHB7CZH7J4VTQ4Q, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.383665245 (631h19m5.25474829s) [noise=true , address=145.79.227.24:4581 , piece_id=LT3US3WZ2XENH5ZLXAVKJUZ2EG5NVSV3KQ4JLEFQ7BN6AHAX7V7A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.387381551 (631h19m5.258464586s) [noise=true , address=145.79.227.48:4301 , piece_id=J4CARGOH7HLJBNPBBS7BU7SYJJQLXE7SHS7T7GWQAQL7VYQYA55A, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.399828752 (631h19m5.270911797s) [noise=true , address=145.79.227.13:4804 , piece_id=H7U2IGOXUTCDRCGRKJPP6IKJOUQQAH6BVBH6QCNJVTH7YAYATCYQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.401727036 (631h19m5.272810081s) [noise=true , address=145.79.227.23:4547 , piece_id=WWTE73A3Z2R7PNAUH6UMQ5XIRNIZAMNDMUEBO3JIHBMCE25NVHZA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.414253538 (631h19m5.285336583s) [noise=true , address=145.79.227.41:4086 , piece_id=TEKGJJFOYC7XARVMBC4CPOQLOO64ISRX4A5NMFJQ4MLXIMJ6H6KQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.572668715 (631h19m5.44375177s) [noise=true , address=77.90.31.13:4626 , piece_id=AKONVNJSNNXP35MCKCUB27NGCLQYYBURQ6TR3Y2GUCNOV6IYSTNA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.607766795 (631h19m5.47884983s) [noise=true , address=145.79.227.12:4765 , piece_id=HLVU4I7L2LUGXTS7CIUUENUSL42PFQ2GEAQJCZN74NA7TRVGBJQA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.616915736 (631h19m5.487998781s) [noise=true , address=77.90.31.21:4877 , piece_id=XBSPVFUBD5Y4N66BTDIG3IUJIT4DBE5RAKULB7FU3EN5XMTVEWRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649284258 (631h19m5.520367293s) [noise=true , address=77.90.31.15:4700 , piece_id=WV6SUWZ5W44KNUU3E3STRMREBIYXFOFLSTGGXAV4DUTMODQ4XVGQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.649706061 (631h19m5.520789096s) [noise=true , address=77.90.31.16:4736 , piece_id=ZMPDPC3VJE2OZMNYGF22BXHOFADQ6PYZ5UDYSZTVA2OR75NN3KAA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.706224688 (631h19m5.577307723s) [noise=true , address=45.91.116.14:4191 , piece_id=2YAJFXX6ME3D4VOOSPTBNGE5GPSIZYLHNZXAOKMQC6PNR73CX6YA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807109562 (631h19m5.678192607s) [noise=true , address=45.91.116.10:4082 , piece_id=QJB3LRLRTXUEJW7THD7U5BBCTDBZ4WO3KMC74ZCIRVVKV7OLDZJQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: <nil> Dec 31 22:48:05.807137545 (631h19m5.67822059s) [seg_pos={0 0}, upload=553209]: Segment reached optimal threshold of 13 pieces. Dec 31 22:48:05.807447786 (631h19m5.678530821s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42360->45.91.116.7:4010: use of closed network connection Dec 31 22:48:05.807460270 (631h19m5.678543305s) [noise=true , address=45.91.116.7:4010 , piece_id=KRCIWPSN7QVUDTELAA2U5IXJANVJWN3Q6RW7EJI6Q2WFAQVODBRQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807583044 (631h19m5.678666089s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807603242 (631h19m5.678686297s) [noise=true , address=145.79.227.40:4051 , piece_id=O7UCGQSW562BO6CGIWIKCOLJVGCZTM454MDABIENSCD5EVGXAUUA, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807788645 (631h19m5.67887169s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 31 22:48:05.807798273 (631h19m5.678881319s) [noise=true , address=145.79.227.44:4195 , piece_id=FS75RLYBXSDELZUWIOIZODGJC7BK6GN5W6DXRBQGZVF6U466RIVQ, seg_pos={0 0}, upload=553209]: Long tail cancellation due to optimal threshold Dec 31 22:48:05.807825135 (631h19m5.67890818s) [seg_pos={0 0}, upload=553209]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:48:05.807931718 (631h19m5.679014763s) [upload=553209]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:48:05.807938711 (631h19m5.679021756s) [upload=553209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:48:06.106235164 (631h19m5.977318199s) [upload=553209]: Done uploading segments... Dec 31 22:48:06.106255563 (631h19m5.977338598s) [upload=553209]: Upload finished. err: <nil> Duration: 30.003214576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XX..................X....................X.........] duration: 6.69462927s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:47:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 6.694654468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XX..................X....................X........] duration: 5.710643297s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:46:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 5.710672304s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XX..................X....................X.......] duration: 25.197711756s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 25.19774025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XX..................X....................X......] duration: 6.65508514s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:45:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 6.65511196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XX..................X....................X.....] duration: 17.921438889s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:44:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 17.921466802s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XX..................X....................X....] duration: 12.537929236s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:43:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 12.537947019s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX..................X....................X...] duration: 6.206792283s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:43:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 6.206818472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX..................X....................X..] duration: 7.454624946s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:42:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 7.454655646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX..................X....................X.] duration: 6.250006581s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:41:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 6.250035437s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.................XX..................X....................X] duration: 30.005539497s trace: https://jaeger.dev.storj.io/trace/532ad8eabcf83c7f FAIL (2025-12-31 22:41:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA) context deadline exceeded ======LOGS====== ments... Dec 31 22:40:37.818329485 (631h11m37.68941252s) [upload=553098]: Got next segment. Inline: false Dec 31 22:40:37.818368640 (631h11m37.689451675s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:40:37.818374110 (631h11m37.689457155s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:40:38.066014213 (631h11m37.937097258s) [seg_pos={0 0}, upload=553098]: Begin upload segment... Dec 31 22:40:38.066065240 (631h11m37.937148275s) [seg_pos={0 0}, upload=553098]: Done begin upload segment. Dec 31 22:40:38.066093805 (631h11m37.93717684s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066199145 (631h11m37.9372822s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066786444 (631h11m37.937869479s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.066961698 (631h11m37.938044743s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067154494 (631h11m37.938237539s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067523257 (631h11m37.938606302s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067563523 (631h11m37.938646558s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.067933338 (631h11m37.939016383s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068028007 (631h11m37.939111062s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068348347 (631h11m37.939431392s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068626447 (631h11m37.939709492s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.068718402 (631h11m37.939801437s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069083548 (631h11m37.940166593s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069168359 (631h11m37.940251404s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.069767279 (631h11m37.940850324s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:38.070338888 (631h11m37.941421923s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Uploading piece... Dec 31 22:40:39.770239349 (631h11m39.641322384s) [upload=553098]: Next returned nil segment Dec 31 22:40:39.770258615 (631h11m39.64134165s) [upload=553098]: Waiting for error group managing segments... Dec 31 22:40:40.056277725 (631h11m39.92736077s) [noise=true , address=145.79.227.38:4979 , piece_id=VQRZOR7NGHSAEOPZ3PR7QGZL7V47QXKY62565EKJAXRB5DELFFLA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.058386020 (631h11m39.929469065s) [noise=true , address=145.79.227.8:4331 , piece_id=FFJ5WXR2Q3NFUQMRGWRCL4MEWVLXFRZUDRPL5QLUN4GAXMBDPGSA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.069875938 (631h11m39.940958973s) [noise=true , address=145.79.227.28:4726 , piece_id=3PJCX34L6LXJAGNQEWOGC7DO2U7TI3J4UAKK3QAWIPO7AO6Q63VA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.073097653 (631h11m39.944180688s) [noise=true , address=145.79.227.27:4689 , piece_id=JHA2HO7GCRKCHGV3U5T5Q5KNPYWJLATRO57B7CMY2AURE4DEBDUQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.076596104 (631h11m39.947679139s) [noise=true , address=145.79.227.41:4087 , piece_id=6NQGHBZRCMM5ZKZXRKV2MICV6LG4RPVUAP2ZECS5DGYKCMZM3MHQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.139890393 (631h11m40.010973428s) [noise=true , address=145.79.227.26:4654 , piece_id=5NC7VOE7VUUB5Y4DPX73S7N7M7WRB2HOTAN65FOURTDNCOT3K34A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143229141 (631h11m40.014312176s) [noise=true , address=145.79.227.45:4230 , piece_id=5DFON3BHDW3YX43SFIP7PS2JX6QFMKENO2WBRDOGQUEB3WWYGNOA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.143689958 (631h11m40.014772993s) [noise=true , address=145.79.227.52:4446 , piece_id=IRN24SJYHZHKP6SVGVSSD26E7AH324KO7TTGTISFFYEL7GUWU7KQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.253182744 (631h11m40.124265779s) [noise=true , address=77.90.31.19:4844 , piece_id=SZAPZVLS4ET2VXDZSW7PHGINWUPFMAMHVXP4SONSDSLPQT5YUNDA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.416020574 (631h11m40.287103609s) [noise=true , address=45.91.116.4:4937 , piece_id=ZNGBHYHXN727TOQL6R6INZP6RVIBAAFVHRDIH4DOVY2LEZSY7AKQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.421830377 (631h11m40.292913412s) [noise=true , address=77.90.31.13:4629 , piece_id=JBEGCXU3OK52PWQRQKLREDJJYPVEJ6US67IUNGJAC4EGJJ5UCVJQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.432667372 (631h11m40.303750407s) [noise=true , address=45.91.116.10:4081 , piece_id=CXXBFKTTNQAMTNRR7OB6MXEAN4OP6ACIRXCL4ZKLI5G33DEK4NQQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489180989 (631h11m40.360264054s) [noise=true , address=45.91.116.5:4297 , piece_id=IO74EUNZGI4QIZQKIITVG6J2VOK6ZGRIZOJDGQJMCUSQTG64I5IA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: <nil> Dec 31 22:40:40.489215044 (631h11m40.360298079s) [seg_pos={0 0}, upload=553098]: Segment reached optimal threshold of 13 pieces. Dec 31 22:40:40.489395517 (631h11m40.360478552s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd): protocol: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59050->145.79.227.39:4014: use of closed network connection Dec 31 22:40:40.489402161 (631h11m40.360485196s) [noise=true , address=145.79.227.39:4014 , piece_id=FQBNQ6MAYYSXEZZFLRHDI6G2GMLX7U7AMQRJV35HBBL5CMHOMURA, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489446806 (631h11m40.360529851s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47178->45.91.116.6:4975: use of closed network connection Dec 31 22:40:40.489457606 (631h11m40.360540661s) [noise=true , address=45.91.116.6:4975 , piece_id=TDY5EDTMTKXFOYFBRUV5IE3AP35CBEBPXNWNPRFVB2CJYSS3SWVQ, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489697212 (631h11m40.360780267s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 31 22:40:40.489706439 (631h11m40.360789485s) [noise=true , address=77.90.31.12:4589 , piece_id=SHWPWK4IUK7VW3TPRYETLAEGU3E73BKKTJACWV4VJX7RUTESJ62A, seg_pos={0 0}, upload=553098]: Long tail cancellation due to optimal threshold Dec 31 22:40:40.489845454 (631h11m40.360928499s) [seg_pos={0 0}, upload=553098]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:40:40.489927651 (631h11m40.361010696s) [upload=553098]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:40:40.489933192 (631h11m40.361016237s) [upload=553098]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:40:40.782715751 (631h11m40.653798796s) [upload=553098]: Done uploading segments... Dec 31 22:40:40.782739675 (631h11m40.653822711s) [upload=553098]: Upload finished. err: <nil> Duration: 30.005560276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.................XX..................X....................] duration: 6.096985184s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:40:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.097013047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.................XX..................X...................] duration: 6.95131375s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:39:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.951340079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.................XX..................X..................] duration: 6.072576757s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.072601824s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.................XX..................X.................] duration: 6.093575728s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:38:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.093592619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.................XX..................X................] duration: 6.346043934s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:36:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.346072999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.................XX..................X...............] duration: 6.16244526s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:36:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.162460939s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.................XX..................X..............] duration: 6.063061209s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:35:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.063088893s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.................XX..................X.............] duration: 6.042284991s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:34:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.042307244s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.................XX..................X............] duration: 14.565578015s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:33:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 14.565608222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.................XX..................X...........] duration: 6.176189817s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:32:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.176236517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.................XX..................X..........] duration: 19.270021342s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:31:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 19.270122746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.................XX..................X.........] duration: 7.289347449s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:30:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 7.289366304s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.................XX..................X........] duration: 27.961263653s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:30:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 27.961292668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.................XX..................X.......] duration: 6.266114576s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.26614295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.................XX..................X......] duration: 6.042222714s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:28:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.042249303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.................XX..................X.....] duration: 6.184524173s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:27:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.184557055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.................XX..................X....] duration: 9.037998118s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:26:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 9.038033425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.................XX..................X...] duration: 6.084444046s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:25:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.084472039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.................XX..................X..] duration: 6.016034532s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 6.016058747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.................XX..................X.] duration: 19.696524285s trace: https://jaeger.dev.storj.io/trace/51ca2eef278825da FAIL (2025-12-31 22:25:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 31 22:24:01.614826990 (630h55m1.485910035s) [upload=552859]: Starting upload Dec 31 22:24:01.614914797 (630h55m1.485997842s) [upload=552859]: Done starting upload Dec 31 22:24:01.615030798 (630h55m1.486113853s) [upload=552859]: Uploading segments... Dec 31 22:24:01.615260004 (630h55m1.486343049s) [upload=552859]: Got next segment. Inline: false Dec 31 22:24:01.615325248 (630h55m1.486408294s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 31 22:24:01.615332602 (630h55m1.486415647s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 31 22:24:01.861544520 (630h55m1.732627566s) [seg_pos={0 0}, upload=552859]: Begin upload segment... Dec 31 22:24:01.861609414 (630h55m1.732692459s) [seg_pos={0 0}, upload=552859]: Done begin upload segment. Dec 31 22:24:01.861655011 (630h55m1.732738066s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861669839 (630h55m1.732752885s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.861923953 (630h55m1.733007008s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.862929518 (630h55m1.734012573s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863076898 (630h55m1.734159943s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863554618 (630h55m1.734637663s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.863888463 (630h55m1.734971508s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.866617680 (630h55m1.737700735s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.867773772 (630h55m1.738856817s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.868614994 (630h55m1.739698039s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.869182514 (630h55m1.740265559s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.870425161 (630h55m1.741508206s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.872828386 (630h55m1.743911431s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.874520528 (630h55m1.745603573s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.878826167 (630h55m1.749909212s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:01.880642956 (630h55m1.751726011s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Uploading piece... Dec 31 22:24:03.488010351 (630h55m3.359093416s) [upload=552859]: Next returned nil segment Dec 31 22:24:03.488028566 (630h55m3.359111611s) [upload=552859]: Waiting for error group managing segments... Dec 31 22:24:03.863957881 (630h55m3.735040927s) [noise=true , address=145.79.227.11:4731 , piece_id=FJYFH36KHQL2QJOQLFWEYGPYUY4AQ6YNGYIRWGY7D7G3HZ72P6KA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.873552412 (630h55m3.744635447s) [noise=true , address=145.79.227.7:4294 , piece_id=ZX55WAKIOGGI7Z3X542T6KNR252I5SM4AMMCE524PW2IGKTNDIWQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.889074187 (630h55m3.760157222s) [noise=true , address=145.79.227.37:4565 , piece_id=FTV7VU2AD6LSJGWKDZWUHBPHNOJIBVUV7PIUF4VYDVBCNPUY5W6Q, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.894913625 (630h55m3.76599667s) [noise=true , address=145.79.227.46:4603 , piece_id=RMSB5TRTRLLZODL5OXMFLJKMH72JBCWKJ2AYBEDGAUYTYBW2AQEQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.902992218 (630h55m3.774075263s) [noise=true , address=145.79.227.20:4438 , piece_id=5SA6EBQXU45GVDR3YAWADJSMZRJR6DHDQOPOKI76U5X3GWBDE3WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.910473683 (630h55m3.781556728s) [noise=true , address=145.79.227.17:4970 , piece_id=2ALPQ45XTTJEE2L6ZZZNF3EWFUV3UMAWD45UHEWCTLIDSVBOFVXQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.917381026 (630h55m3.788464071s) [noise=true , address=145.79.227.3:4150 , piece_id=RZ7RMGU467XYUXVZKNESPZKSSQJLHRUCSVW33FDFSQ66GCLZYAQA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:03.963639328 (630h55m3.834722403s) [noise=true , address=145.79.227.52:4447 , piece_id=FCKZ53SPETL7LG4IIC5X6CZNUKTKSZHHEDBLOOZ6KVYHWMT3GFTA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.064955005 (630h55m3.93603804s) [noise=true , address=77.90.31.14:4664 , piece_id=XTSX637IUTNKWZATZJFJ6J74UNF5PRGW7F4EXLU3GST36VIDTKPA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.151680482 (630h55m4.022763527s) [noise=true , address=77.90.31.11:4555 , piece_id=G4GEFXSFHIE66CTIZQDQNVD3HHJ4ACRG3MCQNDUKZXDKDDPGQOZQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.168985693 (630h55m4.040068738s) [noise=true , address=77.90.31.13:4627 , piece_id=RXU3RMGXKJA7WBTQQQ7UYIEBDQT7OFNW6JRPLPOLIXY6D3VU5GLQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.178717032 (630h55m4.049800067s) [noise=true , address=45.91.116.5:4297 , piece_id=UWMNJWSLWLGKBYQ3SNCUT5MQ5TXPQN3N3FIIV6KI7Z27V6GSYUDA, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179046389 (630h55m4.050129434s) [noise=true , address=77.90.31.15:4697 , piece_id=2BXGBT4IU63MZDAR7WONDNERQW5ERE34WOQX4V35ZKXPDLKAAA4A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: <nil> Dec 31 22:24:04.179068041 (630h55m4.050151086s) [seg_pos={0 0}, upload=552859]: Segment reached optimal threshold of 13 pieces. Dec 31 22:24:04.179277590 (630h55m4.050360625s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179287448 (630h55m4.050370493s) [noise=true , address=45.91.116.6:4975 , piece_id=AE2O5LOAPYF4O23GSUL27S5SWPINEDEOK6WIZGCGCDJD5KPU4H6A, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179363543 (630h55m4.050446578s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179373202 (630h55m4.050456247s) [noise=true , address=45.91.116.8:4045 , piece_id=TOW5R4VLSCTIPXD5OKSQLY6DV4F6L5DH3FMJL7RVLQPQGDE5QDKQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179390955 (630h55m4.050474s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV): protocol: expected piece hash; context canceled; EOF Dec 31 22:24:04.179397958 (630h55m4.050481013s) [noise=true , address=145.79.227.14:4837 , piece_id=6EHIJYYIGSAIPVLT5ZETDVSNJSCDVG5L7PUMPVNXRZNWSEX7O6WQ, seg_pos={0 0}, upload=552859]: Long tail cancellation due to optimal threshold Dec 31 22:24:04.179416714 (630h55m4.05049975s) [seg_pos={0 0}, upload=552859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 31 22:24:04.179517036 (630h55m4.050600081s) [upload=552859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 31 22:24:04.179523919 (630h55m4.050606964s) [upload=552859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 31 22:24:04.467648393 (630h55m4.338731428s) [upload=552859]: Done uploading segments... Dec 31 22:24:04.467674813 (630h55m4.338757858s) [upload=552859]: Upload finished. err: <nil> Duration: 19.696653161s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)