Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 331044 reports (Page 93 of 6621)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X.]
duration: 
5.820168372s
trace: 
https://jaeger.dev.storj.io/trace/725257198532534
FAIL (2025-12-26 19:36:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA) context deadline exceeded
======LOGS======
.BatchRequestItem_ObjectBegin
Dec 26 19:35:32.416073714 (508h6m32.287156749s) [upload=445873]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:35:32.664653082 (508h6m32.535736117s) [seg_pos={0 0}, upload=445873]: Begin upload segment...
Dec 26 19:35:32.664701694 (508h6m32.535784729s) [seg_pos={0 0}, upload=445873]: Done begin upload segment.
Dec 26 19:35:32.664731571 (508h6m32.535814616s) [noise=true , address=145.79.227.36:4519   , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.664920772 (508h6m32.536003817s) [noise=true , address=145.79.227.42:4121   , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.665689175 (508h6m32.53677222s) [noise=true , address=145.79.227.37:4567   , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.666294708 (508h6m32.537377753s) [noise=true , address=145.79.227.45:4231   , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.666986245 (508h6m32.53806928s) [noise=true , address=145.79.227.49:4338   , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667200192 (508h6m32.538283237s) [noise=true , address=145.79.227.3:4151    , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667324149 (508h6m32.538407184s) [noise=true , address=45.91.116.14:4191    , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667694814 (508h6m32.538777859s) [noise=true , address=77.90.31.10:4520     , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667802980 (508h6m32.538886025s) [noise=true , address=77.90.31.21:4880     , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668091610 (508h6m32.539174655s) [noise=true , address=145.79.227.46:4603   , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668459841 (508h6m32.539542896s) [noise=true , address=45.91.116.5:4297     , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668469298 (508h6m32.539552343s) [noise=true , address=45.91.116.7:4009     , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668790591 (508h6m32.539873626s) [noise=true , address=45.91.116.6:4973     , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.669122512 (508h6m32.540205547s) [noise=true , address=145.79.227.39:4015   , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.669954907 (508h6m32.541037952s) [noise=true , address=45.91.116.12:4226    , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.670094884 (508h6m32.541177919s) [noise=true , address=145.79.227.19:4078   , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:34.369947019 (508h6m34.241030054s) [upload=445873]: Next returned nil segment
Dec 26 19:35:34.369990361 (508h6m34.241073416s) [upload=445873]: Waiting for error group managing segments...
Dec 26 19:35:34.658250368 (508h6m34.529333403s) [noise=true , address=145.79.227.46:4603   , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.665658745 (508h6m34.53674178s) [noise=true , address=145.79.227.49:4338   , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.665700594 (508h6m34.536783639s) [noise=true , address=145.79.227.42:4121   , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.670607338 (508h6m34.541690383s) [noise=true , address=145.79.227.39:4015   , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.692231203 (508h6m34.563314248s) [noise=true , address=145.79.227.37:4567   , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.697176921 (508h6m34.568259967s) [noise=true , address=145.79.227.36:4519   , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.705259562 (508h6m34.576342607s) [noise=true , address=145.79.227.45:4231   , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.718464647 (508h6m34.589547692s) [noise=true , address=145.79.227.19:4078   , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.899524947 (508h6m34.770607982s) [noise=true , address=77.90.31.10:4520     , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.939433770 (508h6m34.810516805s) [noise=true , address=45.91.116.6:4973     , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.957276556 (508h6m34.828359601s) [noise=true , address=45.91.116.12:4226    , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.966613435 (508h6m34.83769647s) [noise=true , address=45.91.116.7:4009     , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.970246202 (508h6m34.841329247s) [noise=true , address=77.90.31.21:4880     , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.970260069 (508h6m34.841343104s) [seg_pos={0 0}, upload=445873]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:35:34.970541395 (508h6m34.84162443s) [noise=true , address=45.91.116.5:4297     , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection
Dec 26 19:35:34.970557565 (508h6m34.8416406s) [noise=true , address=45.91.116.5:4297     , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold
Dec 26 19:35:34.970570791 (508h6m34.841653826s) [noise=true , address=45.91.116.14:4191    , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection
Dec 26 19:35:34.970577363 (508h6m34.841660398s) [noise=true , address=45.91.116.14:4191    , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold
Dec 26 19:35:34.970743881 (508h6m34.841826956s) [noise=true , address=145.79.227.3:4151    , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection
Dec 26 19:35:34.970755472 (508h6m34.841838517s) [noise=true , address=145.79.227.3:4151    , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold
Dec 26 19:35:34.970837137 (508h6m34.841920173s) [seg_pos={0 0}, upload=445873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:35:34.970917782 (508h6m34.842000817s) [upload=445873]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:35:34.970922801 (508h6m34.842005836s) [upload=445873]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:35:35.274411846 (508h6m35.145494891s) [upload=445873]: Done uploading segments...
Dec 26 19:35:35.274430442 (508h6m35.145513487s) [upload=445873]: Upload finished. err: <nil>
Duration: 5.820194131s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......X]
duration: 
30.004382056s
trace: 
https://jaeger.dev.storj.io/trace/725257198532534
FAIL (2025-12-26 19:36:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA) context deadline exceeded
======LOGS======
.BatchRequestItem_ObjectBegin
Dec 26 19:35:32.416073714 (508h6m32.287156749s) [upload=445873]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:35:32.664653082 (508h6m32.535736117s) [seg_pos={0 0}, upload=445873]: Begin upload segment...
Dec 26 19:35:32.664701694 (508h6m32.535784729s) [seg_pos={0 0}, upload=445873]: Done begin upload segment.
Dec 26 19:35:32.664731571 (508h6m32.535814616s) [noise=true , address=145.79.227.36:4519   , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.664920772 (508h6m32.536003817s) [noise=true , address=145.79.227.42:4121   , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.665689175 (508h6m32.53677222s) [noise=true , address=145.79.227.37:4567   , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.666294708 (508h6m32.537377753s) [noise=true , address=145.79.227.45:4231   , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.666986245 (508h6m32.53806928s) [noise=true , address=145.79.227.49:4338   , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667200192 (508h6m32.538283237s) [noise=true , address=145.79.227.3:4151    , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667324149 (508h6m32.538407184s) [noise=true , address=45.91.116.14:4191    , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667694814 (508h6m32.538777859s) [noise=true , address=77.90.31.10:4520     , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.667802980 (508h6m32.538886025s) [noise=true , address=77.90.31.21:4880     , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668091610 (508h6m32.539174655s) [noise=true , address=145.79.227.46:4603   , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668459841 (508h6m32.539542896s) [noise=true , address=45.91.116.5:4297     , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668469298 (508h6m32.539552343s) [noise=true , address=45.91.116.7:4009     , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.668790591 (508h6m32.539873626s) [noise=true , address=45.91.116.6:4973     , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.669122512 (508h6m32.540205547s) [noise=true , address=145.79.227.39:4015   , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.669954907 (508h6m32.541037952s) [noise=true , address=45.91.116.12:4226    , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:32.670094884 (508h6m32.541177919s) [noise=true , address=145.79.227.19:4078   , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Uploading piece...
Dec 26 19:35:34.369947019 (508h6m34.241030054s) [upload=445873]: Next returned nil segment
Dec 26 19:35:34.369990361 (508h6m34.241073416s) [upload=445873]: Waiting for error group managing segments...
Dec 26 19:35:34.658250368 (508h6m34.529333403s) [noise=true , address=145.79.227.46:4603   , piece_id=TNZB76INU3PMB6SWG5DBJMR47Q2P2J7DF2XXDN4IGSFATIQXTIJQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.665658745 (508h6m34.53674178s) [noise=true , address=145.79.227.49:4338   , piece_id=NLBLJ5SWF67MRFTMMSH3EUMHFIGWZIHNKU364CIMPNLVQISHDKAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.665700594 (508h6m34.536783639s) [noise=true , address=145.79.227.42:4121   , piece_id=6K7CZHGNWTECDT2LN4TPJQNHXD7C74T2VFIYZR35PICZU35D5PEA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.670607338 (508h6m34.541690383s) [noise=true , address=145.79.227.39:4015   , piece_id=LBMAA4YAAMH532P6YGYPNQGHXOPNURFN3FA2I4CORCD77QNSK2AQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.692231203 (508h6m34.563314248s) [noise=true , address=145.79.227.37:4567   , piece_id=GEATQYGSNS55Q6KNBTTW36PMZHCGB4FYMQMJJ7AWFIZNVZAZBH5A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.697176921 (508h6m34.568259967s) [noise=true , address=145.79.227.36:4519   , piece_id=BYK3O6GOUHCVRS4DPOICNMYX7DLVMWY4G6RRPBFFEZUXOHXL6H4A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.705259562 (508h6m34.576342607s) [noise=true , address=145.79.227.45:4231   , piece_id=DTL5UZIOTCGIGIGJIE77M2J2USY4CRUGV4B4I5IVE2JCJHOPHEZQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.718464647 (508h6m34.589547692s) [noise=true , address=145.79.227.19:4078   , piece_id=NLOFZDPLKEVLD53Q4ILQM7AX3NCI7IE6N2X5M2HWCKRN5GEN3AAA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.899524947 (508h6m34.770607982s) [noise=true , address=77.90.31.10:4520     , piece_id=E26JISLDLMD6PP2AAGIOOWOMELCOLTC4464QFDE4V42XACJITQTQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.939433770 (508h6m34.810516805s) [noise=true , address=45.91.116.6:4973     , piece_id=YD2I64QLIP5XRIQYWMKWPM2YQFLUL6YZRQXHBEYYLGYFFZPJABNQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.957276556 (508h6m34.828359601s) [noise=true , address=45.91.116.12:4226    , piece_id=BXODDPS5UUASMZ47A4EVF5WQFNYXUE7MDOBMEJLS2KVBOWMU5VNA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.966613435 (508h6m34.83769647s) [noise=true , address=45.91.116.7:4009     , piece_id=7GWILCYXB3ACUFBECP5QHOXLGLTMNPWHMBAHTAUM44QAJ4Z5WNIQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.970246202 (508h6m34.841329247s) [noise=true , address=77.90.31.21:4880     , piece_id=LTNES4PVCDDUNK7IOM2AKAAKYYX5IMVF3OQ725XVCJVYNODEAE6A, seg_pos={0 0}, upload=445873]: Done uploading piece. err: <nil>
Dec 26 19:35:34.970260069 (508h6m34.841343104s) [seg_pos={0 0}, upload=445873]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:35:34.970541395 (508h6m34.84162443s) [noise=true , address=45.91.116.5:4297     , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53450->45.91.116.5:4297: use of closed network connection
Dec 26 19:35:34.970557565 (508h6m34.8416406s) [noise=true , address=45.91.116.5:4297     , piece_id=6RH535L2PIT7ZYSHEDZSUOBEUMPD7CQFW6PANPYT4BGHZLVLOAYA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold
Dec 26 19:35:34.970570791 (508h6m34.841653826s) [noise=true , address=45.91.116.14:4191    , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53592->45.91.116.14:4191: use of closed network connection
Dec 26 19:35:34.970577363 (508h6m34.841660398s) [noise=true , address=45.91.116.14:4191    , piece_id=N34QQPNGIGEGXALCHX2PC63XUIKTJCQJRRZWQ5JUICH5Y6HUY2SQ, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold
Dec 26 19:35:34.970743881 (508h6m34.841826956s) [noise=true , address=145.79.227.3:4151    , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53690->145.79.227.3:4151: use of closed network connection
Dec 26 19:35:34.970755472 (508h6m34.841838517s) [noise=true , address=145.79.227.3:4151    , piece_id=U4JZLWHDJEQFHB7UJWHIN2NF5M7UL4M3TSBI2AOT2BVYVDAN73KA, seg_pos={0 0}, upload=445873]: Long tail cancellation due to optimal threshold
Dec 26 19:35:34.970837137 (508h6m34.841920173s) [seg_pos={0 0}, upload=445873]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:35:34.970917782 (508h6m34.842000817s) [upload=445873]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:35:34.970922801 (508h6m34.842005836s) [upload=445873]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:35:35.274411846 (508h6m35.145494891s) [upload=445873]: Done uploading segments...
Dec 26 19:35:35.274430442 (508h6m35.145513487s) [upload=445873]: Upload finished. err: <nil>
Duration: 30.004409448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX......]
duration: 
7.306531825s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:34:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 7.306563886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX.....]
duration: 
7.206037947s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:33:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 7.206065239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX....]
duration: 
6.433736853s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:32:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 6.433753385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX...]
duration: 
6.296305188s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:32:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 6.296335246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX..]
duration: 
12.866996317s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:31:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 12.867014661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX.]
duration: 
7.687812027s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:30:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 7.687832938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXXX]
duration: 
30.002706126s
trace: 
https://jaeger.dev.storj.io/trace/39a64a2b3969bb0b
FAIL (2025-12-26 19:30:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: 4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ) context deadline exceeded
======LOGS======
Dec 26 19:30:13.787517519 (508h1m13.658600554s) [upload=445798]: Starting upload
Dec 26 19:30:13.787603884 (508h1m13.658686949s) [upload=445798]: Done starting upload
Dec 26 19:30:13.787841155 (508h1m13.6589242s) [upload=445798]: Uploading segments...
Dec 26 19:30:13.787940714 (508h1m13.659023759s) [upload=445798]: Got next segment. Inline: false
Dec 26 19:30:13.787988536 (508h1m13.659071571s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:30:13.787994577 (508h1m13.659077622s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:30:14.036843449 (508h1m13.907926484s) [seg_pos={0 0}, upload=445798]: Begin upload segment...
Dec 26 19:30:14.036895938 (508h1m13.907978973s) [seg_pos={0 0}, upload=445798]: Done begin upload segment.
Dec 26 19:30:14.036922649 (508h1m13.908005685s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.037146657 (508h1m13.908229702s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038042121 (508h1m13.909125156s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038145307 (508h1m13.909228343s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038388961 (508h1m13.909472007s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038607689 (508h1m13.909690734s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.038807639 (508h1m13.909890674s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039176071 (508h1m13.910259106s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039345584 (508h1m13.910428629s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039538260 (508h1m13.910621305s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039925337 (508h1m13.911008372s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.039955955 (508h1m13.91103899s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040290272 (508h1m13.911373307s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.040542653 (508h1m13.911625688s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041077752 (508h1m13.912160787s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:14.041523931 (508h1m13.912606966s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Uploading piece...
Dec 26 19:30:15.651632940 (508h1m15.522715975s) [upload=445798]: Next returned nil segment
Dec 26 19:30:15.651658879 (508h1m15.522741924s) [upload=445798]: Waiting for error group managing segments...
Dec 26 19:30:16.030957340 (508h1m15.902040385s) [noise=true , address=145.79.227.46:4601   , piece_id=YBAXZM6ANRGDZ4VYAKAQ7ZE5EEU2XVBWN4TVWEJXH2PHZLSK5KBQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.033473150 (508h1m15.904556185s) [noise=true , address=145.79.227.52:4447   , piece_id=7YPMDX3YWX4IGITCQUIE56SUEKWGQFRESWLRUKWDGVL6PSQZGJ6Q, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.039516378 (508h1m15.910599423s) [noise=true , address=145.79.227.11:4730   , piece_id=HHODSJ4CHFPA3ZICGP42A7SBBEPZOL7IGCM2LVUBC3GZF34LCWHA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.040807516 (508h1m15.911890551s) [noise=true , address=145.79.227.45:4229   , piece_id=UBOZ2VYJMFASJ5Y3VBSOB2FXYJ5UDXBAOPTHNNT747U5WKEH4RLA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.044564951 (508h1m15.915647986s) [noise=true , address=145.79.227.36:4518   , piece_id=U3SE2PVWDFBUIPOXBLNB5ZR7TEKPAVB246NFKVBHD2GNDIKCERIA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.051106367 (508h1m15.922189402s) [noise=true , address=145.79.227.47:4265   , piece_id=N7Q2EVWMWJX36NTBWWCUY7UR6O4BRRQFKJVIYNLIJKWF2O7VDBFA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.058618822 (508h1m15.929701857s) [noise=true , address=145.79.227.48:4301   , piece_id=IX6725XHRANQHDVCB4WVRSKQPITJ3FSNWGC7VJKFZPKJWHT3K6PQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.069751150 (508h1m15.940834185s) [noise=true , address=145.79.227.49:4338   , piece_id=LILQHGDOGK5FNOIWH2CSX6HZ76E2Q6ILXIDCPTPZ6Z6B6SO437EA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.073068076 (508h1m15.944151111s) [noise=true , address=145.79.227.42:4122   , piece_id=EXH4MMIN7CHCM6OILZWXIM4JNB6IZOMDHRGOOOGXXVKXWEYL6J6A, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.076405150 (508h1m15.947488186s) [noise=true , address=145.79.227.37:4565   , piece_id=F4RQDXPYI2MCN4RJZMYJT4EUGTT7J5SU7UMALU3ATNP6ZP2Y4NKA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.227562450 (508h1m16.098645495s) [noise=true , address=77.90.31.21:4880     , piece_id=JAB5L4J5IEYKIHEE6VGBKDCNGWCHHHSNPGBUTQ5J7K3ZKI2UGREA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.288235873 (508h1m16.159318918s) [noise=true , address=45.91.116.4:4937     , piece_id=5K2HD5XC6UOMRRBRSE5U2NO47NHJ5ETVR63ABDOZRNDZ56TTBCQA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333361400 (508h1m16.204444435s) [noise=true , address=45.91.116.8:4046     , piece_id=4SO3BWBDC7PVI5XLCRS7OQCZ6RF62BVRNPHVSPGN2QTFXXZXPPCQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: <nil>
Dec 26 19:30:16.333389464 (508h1m16.204472509s) [seg_pos={0 0}, upload=445798]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:30:16.333649959 (508h1m16.204733004s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333659537 (508h1m16.204742592s) [noise=true , address=45.91.116.6:4975     , piece_id=WYOV47W45J7OIJNAE6KDAKTP4XAGHW3YDWETTLR45VIYTUPHX5JQ, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.333858546 (508h1m16.204941591s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:30:16.333874617 (508h1m16.204957662s) [noise=true , address=145.79.227.17:4972   , piece_id=GEEPYFADUWHFP6CNUY2XP2Z3P2E534X645JH6RSCBTCYGVDKNQEA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334345694 (508h1m16.205428729s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:45368->145.79.227.41:4085: use of closed network connection
Dec 26 19:30:16.334366885 (508h1m16.20544993s) [noise=true , address=145.79.227.41:4085   , piece_id=V7VSOEB7TIYHYK3NBKYJ5DQIWBE42BFKT53RSNMEMNBQ54SMQ7GA, seg_pos={0 0}, upload=445798]: Long tail cancellation due to optimal threshold
Dec 26 19:30:16.334457336 (508h1m16.205540381s) [seg_pos={0 0}, upload=445798]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:30:16.334637379 (508h1m16.205720424s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:30:16.334646958 (508h1m16.205730003s) [upload=445798]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:30:16.628885289 (508h1m16.499968324s) [upload=445798]: Done uploading segments...
Dec 26 19:30:16.628906248 (508h1m16.499989283s) [upload=445798]: Upload finished. err: <nil>
Duration: 30.002732987s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXXX]
duration: 
30.005725155s
trace: 
https://jaeger.dev.storj.io/trace/3daee636559e1cec
FAIL (2025-12-26 19:29:52 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: DJYANR5BB6EYTKHHZGJMJF3XZ6RTVYSDPXL2TAOEP7BRFHHCLTDA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: HZTOKQFGOC7PZ6PNE2AEI4B7ZPLV4KWZVQRTR7L5TRA4UHP75HBQ) context deadline exceeded
======LOGS======
Dec 26 19:29:23.427370966 (508h0m23.298454001s) [upload=445785]: Starting upload
Dec 26 19:29:23.427429568 (508h0m23.298512603s) [upload=445785]: Done starting upload
Dec 26 19:29:23.427605463 (508h0m23.298688508s) [upload=445785]: Uploading segments...
Dec 26 19:29:23.427641962 (508h0m23.298724997s) [upload=445785]: Got next segment. Inline: false
Dec 26 19:29:23.427668303 (508h0m23.298751338s) [upload=445785]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:29:23.427672671 (508h0m23.298755716s) [upload=445785]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:29:23.685240285 (508h0m23.55632333s) [seg_pos={0 0}, upload=445785]: Begin upload segment...
Dec 26 19:29:23.685284329 (508h0m23.556367364s) [seg_pos={0 0}, upload=445785]: Done begin upload segment.
Dec 26 19:29:23.685309527 (508h0m23.556392562s) [noise=true , address=145.79.227.4:4187    , piece_id=D3LTSZXDLLQEJ2JUMK4TTXMUAKHCHZDJYGZY6ZO6CTMG7U5SFWEQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.689883055 (508h0m23.56096612s) [noise=true , address=145.79.227.21:4475   , piece_id=ABYAXLZPRMVDYX2PZRW7XBGPFXLFKFFXMZ4TLNQWWHEYYJ4AXJQA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.691775148 (508h0m23.562858193s) [noise=true , address=145.79.227.45:4230   , piece_id=TUHJXV5ZHD34IFRW7EH7HCCKOCFDHG7TQMGKAKNHAPWYVYVXVCGQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.693130779 (508h0m23.564213824s) [noise=true , address=145.79.227.11:4730   , piece_id=ZTS3PEQA46YHDBVAHLSEQMIPDDBOIZBJVW4HCA3WJW5F5BZTO5YA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.694416928 (508h0m23.565499963s) [noise=true , address=77.90.31.18:4808     , piece_id=DJYANR5BB6EYTKHHZGJMJF3XZ6RTVYSDPXL2TAOEP7BRFHHCLTDA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.695628095 (508h0m23.56671114s) [noise=true , address=145.79.227.36:4519   , piece_id=72W22DPNORECKX3GRYHK5ZS4FJD4HMJ724OSU4KK43WSKTAVHUFQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.696697942 (508h0m23.567780987s) [noise=true , address=145.79.227.49:4337   , piece_id=RNZAWUNENGDCF33JXS3CIP33X25LRCLX5E2MWRFA6H4SQIEAIVPA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.697948624 (508h0m23.569031659s) [noise=true , address=45.91.116.5:4299     , piece_id=HZTOKQFGOC7PZ6PNE2AEI4B7ZPLV4KWZVQRTR7L5TRA4UHP75HBQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.699233470 (508h0m23.570316506s) [noise=true , address=145.79.227.39:4014   , piece_id=3IA5BCBNUHGPCDJ6HWNCCUNWQQ4TUTGIPC6LVPAWI3Y3RMFA33GQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.700512255 (508h0m23.57159529s) [noise=true , address=145.79.227.38:4978   , piece_id=CAWZ72QZ34FFTX3WXJJB4EAWN4UWGE7QUNHRRRW2PGV26FIRZU2A, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.701856595 (508h0m23.57293963s) [noise=true , address=45.91.116.9:4264     , piece_id=5LBGA57TPBIL7S43JI2YAKQP65Y5WFUIZC7N23PDGTXPI3M6ATXA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.703068703 (508h0m23.574151738s) [noise=true , address=145.79.227.50:4374   , piece_id=F2TUD7D625NQKKFORBV2CR3CROWBEM5NACK4BTR3Z5NXZMMEU5RQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.704257246 (508h0m23.575340281s) [noise=true , address=145.79.227.46:4602   , piece_id=TA4PJFFXWHUJJMAR4OGTW3F5FI7VDFZL2TRZINWEDNBCHCXIHY3Q, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.705582339 (508h0m23.576665395s) [noise=true , address=145.79.227.52:4447   , piece_id=UBGKPT6WYNL7RSD4FFAJIALSTQQW5CB46U2YEVWPIQJNWQFYXAGA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.706972316 (508h0m23.578055351s) [noise=true , address=77.90.31.10:4520     , piece_id=LKQSJOYBOJDHMIS3CP6LE3XIX2AL5ZCJQDO37SIP5A54I7XCOWOQ, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:23.708149258 (508h0m23.579232293s) [noise=true , address=45.91.116.14:4192    , piece_id=3SZRRHSKLACT6BU6D7XHMEUB5IE44LBZEYBGZSLXIN5QIMJSC2ZA, seg_pos={0 0}, upload=445785]: Uploading piece...
Dec 26 19:29:25.411769017 (508h0m25.282852052s) [upload=445785]: Next returned nil segment
Dec 26 19:29:25.411811608 (508h0m25.282894643s) [upload=445785]: Waiting for error group managing segments...
Dec 26 19:29:25.698202356 (508h0m25.569285391s) [noise=true , address=145.79.227.21:4475   , piece_id=ABYAXLZPRMVDYX2PZRW7XBGPFXLFKFFXMZ4TLNQWWHEYYJ4AXJQA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.698434057 (508h0m25.569517092s) [noise=true , address=145.79.227.52:4447   , piece_id=UBGKPT6WYNL7RSD4FFAJIALSTQQW5CB46U2YEVWPIQJNWQFYXAGA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.699301128 (508h0m25.570384163s) [noise=true , address=145.79.227.45:4230   , piece_id=TUHJXV5ZHD34IFRW7EH7HCCKOCFDHG7TQMGKAKNHAPWYVYVXVCGQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.714752430 (508h0m25.585835465s) [noise=true , address=145.79.227.46:4602   , piece_id=TA4PJFFXWHUJJMAR4OGTW3F5FI7VDFZL2TRZINWEDNBCHCXIHY3Q, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.720797712 (508h0m25.591880757s) [noise=true , address=145.79.227.50:4374   , piece_id=F2TUD7D625NQKKFORBV2CR3CROWBEM5NACK4BTR3Z5NXZMMEU5RQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.732212037 (508h0m25.603295072s) [noise=true , address=145.79.227.39:4014   , piece_id=3IA5BCBNUHGPCDJ6HWNCCUNWQQ4TUTGIPC6LVPAWI3Y3RMFA33GQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.732631846 (508h0m25.603714891s) [noise=true , address=145.79.227.36:4519   , piece_id=72W22DPNORECKX3GRYHK5ZS4FJD4HMJ724OSU4KK43WSKTAVHUFQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.735263757 (508h0m25.606346802s) [noise=true , address=145.79.227.38:4978   , piece_id=CAWZ72QZ34FFTX3WXJJB4EAWN4UWGE7QUNHRRRW2PGV26FIRZU2A, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.743327212 (508h0m25.614410257s) [noise=true , address=145.79.227.49:4337   , piece_id=RNZAWUNENGDCF33JXS3CIP33X25LRCLX5E2MWRFA6H4SQIEAIVPA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.763319370 (508h0m25.634402415s) [noise=true , address=145.79.227.4:4187    , piece_id=D3LTSZXDLLQEJ2JUMK4TTXMUAKHCHZDJYGZY6ZO6CTMG7U5SFWEQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:25.964663325 (508h0m25.83574636s) [noise=true , address=145.79.227.11:4730   , piece_id=ZTS3PEQA46YHDBVAHLSEQMIPDDBOIZBJVW4HCA3WJW5F5BZTO5YA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:26.007126412 (508h0m25.878209447s) [noise=true , address=77.90.31.18:4808     , piece_id=DJYANR5BB6EYTKHHZGJMJF3XZ6RTVYSDPXL2TAOEP7BRFHHCLTDA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:26.101332597 (508h0m25.972415632s) [noise=true , address=45.91.116.5:4299     , piece_id=HZTOKQFGOC7PZ6PNE2AEI4B7ZPLV4KWZVQRTR7L5TRA4UHP75HBQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: <nil>
Dec 26 19:29:26.101357664 (508h0m25.972440709s) [seg_pos={0 0}, upload=445785]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:29:26.101591310 (508h0m25.972674345s) [noise=true , address=45.91.116.14:4192    , piece_id=3SZRRHSKLACT6BU6D7XHMEUB5IE44LBZEYBGZSLXIN5QIMJSC2ZA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 26 19:29:26.101599435 (508h0m25.97268247s) [noise=true , address=45.91.116.14:4192    , piece_id=3SZRRHSKLACT6BU6D7XHMEUB5IE44LBZEYBGZSLXIN5QIMJSC2ZA, seg_pos={0 0}, upload=445785]: Long tail cancellation due to optimal threshold
Dec 26 19:29:26.101772885 (508h0m25.97285593s) [noise=true , address=45.91.116.9:4264     , piece_id=5LBGA57TPBIL7S43JI2YAKQP65Y5WFUIZC7N23PDGTXPI3M6ATXA, seg_pos={0 0}, upload=445785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:47768->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:47768->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47768->45.91.116.9:4264: use of closed network connection
Dec 26 19:29:26.101780570 (508h0m25.972863615s) [noise=true , address=45.91.116.9:4264     , piece_id=5LBGA57TPBIL7S43JI2YAKQP65Y5WFUIZC7N23PDGTXPI3M6ATXA, seg_pos={0 0}, upload=445785]: Long tail cancellation due to optimal threshold
Dec 26 19:29:26.101975291 (508h0m25.973058346s) [noise=true , address=77.90.31.10:4520     , piece_id=LKQSJOYBOJDHMIS3CP6LE3XIX2AL5ZCJQDO37SIP5A54I7XCOWOQ, seg_pos={0 0}, upload=445785]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 26 19:29:26.101982624 (508h0m25.97306567s) [noise=true , address=77.90.31.10:4520     , piece_id=LKQSJOYBOJDHMIS3CP6LE3XIX2AL5ZCJQDO37SIP5A54I7XCOWOQ, seg_pos={0 0}, upload=445785]: Long tail cancellation due to optimal threshold
Dec 26 19:29:26.102003565 (508h0m25.97308661s) [seg_pos={0 0}, upload=445785]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:29:26.102101672 (508h0m25.973184717s) [upload=445785]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:29:26.102108585 (508h0m25.97319163s) [upload=445785]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:29:26.397196714 (508h0m26.268279749s) [upload=445785]: Done uploading segments...
Dec 26 19:29:26.397215139 (508h0m26.268298184s) [upload=445785]: Upload finished. err: <nil>
Duration: 30.005742919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [..X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XXX]
duration: 
30.00440889s
trace: 
https://jaeger.dev.storj.io/trace/24c8fc16133b06c
FAIL (2025-12-26 19:28:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: KW5CPEMPKGPC4OGKTRSUQT6FNY3JPY4COBMNJIWI7RJI4CW3KUUQ) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: UOBAGMEZ5AU7JTV53WPRDF543MO4ZC6L6RU4VCLBPZOUEDUIW5MA) context deadline exceeded
======LOGS======
Dec 26 19:28:20.086996090 (507h59m19.958079135s) [upload=445770]: Starting upload
Dec 26 19:28:20.087073647 (507h59m19.958156683s) [upload=445770]: Done starting upload
Dec 26 19:28:20.087268749 (507h59m19.958351784s) [upload=445770]: Uploading segments...
Dec 26 19:28:20.087304517 (507h59m19.958387553s) [upload=445770]: Got next segment. Inline: false
Dec 26 19:28:20.087344824 (507h59m19.958427859s) [upload=445770]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:28:20.087349112 (507h59m19.958432147s) [upload=445770]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:28:20.347294434 (507h59m20.21837747s) [seg_pos={0 0}, upload=445770]: Begin upload segment...
Dec 26 19:28:20.347343347 (507h59m20.218426382s) [seg_pos={0 0}, upload=445770]: Done begin upload segment.
Dec 26 19:28:20.347376040 (507h59m20.218459075s) [noise=true , address=145.79.227.52:4447   , piece_id=A4KHWZ6SNPW5UABWXV224H6FJQQIKQEMNXGLAQIR2TCVRNP7D55A, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.348306672 (507h59m20.219389717s) [noise=true , address=145.79.227.41:4087   , piece_id=OT3PLCULCUEUMBXALNGP4JVIENRKUTZ76A3S3YH6WPUSSTSSSIMQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.348759534 (507h59m20.219842569s) [noise=true , address=45.91.116.7:4011     , piece_id=3JCPE5GG6XH4P4NTSUYTMMZYPOS4RX2FOD3KL2XRY6GD2UU7UCLA, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.349123998 (507h59m20.220207043s) [noise=true , address=145.79.227.36:4517   , piece_id=IK6HZP4TGLKUF45T3D33RB3IZO65RVGX55I5Y5CCCQUPYKD4EYMQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.349411435 (507h59m20.22049448s) [noise=true , address=45.91.116.10:4084    , piece_id=PJRZRVZFSM63OKSAQKNV2VHR5RBVGF2VWLJ7KL46ONP2CZQRVLSA, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.349477581 (507h59m20.220560616s) [noise=true , address=45.91.116.4:4939     , piece_id=2W6P7RHOX3ZI2ANYDXCGB44WKHA7C7KE5WPMPF6GRCPCESCFZBAQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.349885448 (507h59m20.220968483s) [noise=true , address=77.90.31.17:4773     , piece_id=XYKHLFRJOEVB2T654G6EODXUN5NX7WGLJZ4RZDFUDF2FRK526FQQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.350258659 (507h59m20.221341704s) [noise=true , address=45.91.116.11:4117    , piece_id=KW5CPEMPKGPC4OGKTRSUQT6FNY3JPY4COBMNJIWI7RJI4CW3KUUQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.350301380 (507h59m20.221384425s) [noise=true , address=77.90.31.19:4844     , piece_id=A5ALTTDV6T2EIOBVDV2ASHS2ZIE6B6BNKM6VLZXCCE54DI5JW2RQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.350618474 (507h59m20.221701519s) [noise=true , address=45.91.116.12:4226    , piece_id=DKKVVEMPD6WNDOS76EED4BPMVZMIDQTU7I5CH5453PJQXLOBZ2MQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.351026180 (507h59m20.222109215s) [noise=true , address=145.79.227.51:4411   , piece_id=GGDIWGBFN67LYUPYWOI6TVR3J7TRKWESLWO5IX4AWTGWG2WJQ4EA, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.351403048 (507h59m20.222486083s) [noise=true , address=145.79.227.5:4222    , piece_id=72TPYCQG7FS62LI5TL2DPNUPAJFV7M7LTOT5I2FWMZ7U5P7KNKBQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.351463422 (507h59m20.222546457s) [noise=true , address=45.91.116.6:4976     , piece_id=UOBAGMEZ5AU7JTV53WPRDF543MO4ZC6L6RU4VCLBPZOUEDUIW5MA, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.351836302 (507h59m20.222919338s) [noise=true , address=145.79.227.37:4567   , piece_id=4OLF5PS43TR3D2O2ZWAIQTG5RGPTI5M4HCHQ5R4HPTTYTDHLTHIQ, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.352027998 (507h59m20.223111033s) [noise=true , address=77.90.31.21:4878     , piece_id=27P3UY6S545J4I7DFOWZWSJPFXDPTPFSC3VBJJCV43IMHEBNJQGA, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:20.352277964 (507h59m20.223360999s) [noise=true , address=77.90.31.10:4517     , piece_id=NP6QYUDVAFNBEKYF7VAWMHPWC7GXWXBUY6ESYQED7OASRDFAXE5Q, seg_pos={0 0}, upload=445770]: Uploading piece...
Dec 26 19:28:22.063140836 (507h59m21.934223871s) [upload=445770]: Next returned nil segment
Dec 26 19:28:22.063156916 (507h59m21.934239951s) [upload=445770]: Waiting for error group managing segments...
Dec 26 19:28:22.353113845 (507h59m22.22419688s) [noise=true , address=145.79.227.37:4567   , piece_id=4OLF5PS43TR3D2O2ZWAIQTG5RGPTI5M4HCHQ5R4HPTTYTDHLTHIQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.360348070 (507h59m22.231431105s) [noise=true , address=145.79.227.41:4087   , piece_id=OT3PLCULCUEUMBXALNGP4JVIENRKUTZ76A3S3YH6WPUSSTSSSIMQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.363007053 (507h59m22.234090098s) [noise=true , address=145.79.227.52:4447   , piece_id=A4KHWZ6SNPW5UABWXV224H6FJQQIKQEMNXGLAQIR2TCVRNP7D55A, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.375569715 (507h59m22.24665275s) [noise=true , address=145.79.227.36:4517   , piece_id=IK6HZP4TGLKUF45T3D33RB3IZO65RVGX55I5Y5CCCQUPYKD4EYMQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.375954317 (507h59m22.247037352s) [noise=true , address=145.79.227.51:4411   , piece_id=GGDIWGBFN67LYUPYWOI6TVR3J7TRKWESLWO5IX4AWTGWG2WJQ4EA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.464190494 (507h59m22.33527354s) [noise=true , address=145.79.227.5:4222    , piece_id=72TPYCQG7FS62LI5TL2DPNUPAJFV7M7LTOT5I2FWMZ7U5P7KNKBQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.590783859 (507h59m22.461866894s) [noise=true , address=45.91.116.10:4084    , piece_id=PJRZRVZFSM63OKSAQKNV2VHR5RBVGF2VWLJ7KL46ONP2CZQRVLSA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.596918882 (507h59m22.468001927s) [noise=true , address=45.91.116.11:4117    , piece_id=KW5CPEMPKGPC4OGKTRSUQT6FNY3JPY4COBMNJIWI7RJI4CW3KUUQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.598764727 (507h59m22.469847772s) [noise=true , address=45.91.116.6:4976     , piece_id=UOBAGMEZ5AU7JTV53WPRDF543MO4ZC6L6RU4VCLBPZOUEDUIW5MA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.661889257 (507h59m22.532972292s) [noise=true , address=45.91.116.7:4011     , piece_id=3JCPE5GG6XH4P4NTSUYTMMZYPOS4RX2FOD3KL2XRY6GD2UU7UCLA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.695616008 (507h59m22.566699044s) [noise=true , address=77.90.31.10:4517     , piece_id=NP6QYUDVAFNBEKYF7VAWMHPWC7GXWXBUY6ESYQED7OASRDFAXE5Q, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.700244672 (507h59m22.571327717s) [noise=true , address=45.91.116.4:4939     , piece_id=2W6P7RHOX3ZI2ANYDXCGB44WKHA7C7KE5WPMPF6GRCPCESCFZBAQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.707794629 (507h59m22.578877674s) [noise=true , address=45.91.116.12:4226    , piece_id=DKKVVEMPD6WNDOS76EED4BPMVZMIDQTU7I5CH5453PJQXLOBZ2MQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: <nil>
Dec 26 19:28:22.707819486 (507h59m22.578902531s) [seg_pos={0 0}, upload=445770]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:28:22.708029075 (507h59m22.579112111s) [noise=true , address=77.90.31.17:4773     , piece_id=XYKHLFRJOEVB2T654G6EODXUN5NX7WGLJZ4RZDFUDF2FRK526FQQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 26 19:28:22.708037171 (507h59m22.579120206s) [noise=true , address=77.90.31.17:4773     , piece_id=XYKHLFRJOEVB2T654G6EODXUN5NX7WGLJZ4RZDFUDF2FRK526FQQ, seg_pos={0 0}, upload=445770]: Long tail cancellation due to optimal threshold
Dec 26 19:28:22.708141860 (507h59m22.579224925s) [noise=true , address=77.90.31.19:4844     , piece_id=A5ALTTDV6T2EIOBVDV2ASHS2ZIE6B6BNKM6VLZXCCE54DI5JW2RQ, seg_pos={0 0}, upload=445770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 26 19:28:22.708152390 (507h59m22.579235425s) [noise=true , address=77.90.31.19:4844     , piece_id=A5ALTTDV6T2EIOBVDV2ASHS2ZIE6B6BNKM6VLZXCCE54DI5JW2RQ, seg_pos={0 0}, upload=445770]: Long tail cancellation due to optimal threshold
Dec 26 19:28:22.708283950 (507h59m22.579366996s) [noise=true , address=77.90.31.21:4878     , piece_id=27P3UY6S545J4I7DFOWZWSJPFXDPTPFSC3VBJJCV43IMHEBNJQGA, seg_pos={0 0}, upload=445770]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 26 19:28:22.708292557 (507h59m22.579375602s) [noise=true , address=77.90.31.21:4878     , piece_id=27P3UY6S545J4I7DFOWZWSJPFXDPTPFSC3VBJJCV43IMHEBNJQGA, seg_pos={0 0}, upload=445770]: Long tail cancellation due to optimal threshold
Dec 26 19:28:22.708311754 (507h59m22.579394799s) [seg_pos={0 0}, upload=445770]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:28:22.708452983 (507h59m22.579536018s) [upload=445770]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:28:22.708459495 (507h59m22.57954256s) [upload=445770]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:28:23.007946681 (507h59m22.879029726s) [upload=445770]: Done uploading segments...
Dec 26 19:28:23.007965386 (507h59m22.879048421s) [upload=445770]: Upload finished. err: <nil>
Duration: 30.004432645s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.XX]
duration: 
30.001734416s
trace: 
https://jaeger.dev.storj.io/trace/57629f9909823f
FAIL (2025-12-26 19:28:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 19:27:41.059559220 (507h58m40.930642255s) [upload=445758]: Starting upload
Dec 26 19:27:41.059621878 (507h58m40.930704944s) [upload=445758]: Done starting upload
Dec 26 19:27:41.059724604 (507h58m40.930807639s) [upload=445758]: Uploading segments...
Dec 26 19:27:41.059836047 (507h58m40.930919092s) [upload=445758]: Got next segment. Inline: false
Dec 26 19:27:41.059867397 (507h58m40.930950432s) [upload=445758]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:27:41.059871795 (507h58m40.93095483s) [upload=445758]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:27:41.308684257 (507h58m41.179767293s) [seg_pos={0 0}, upload=445758]: Begin upload segment...
Dec 26 19:27:41.308727480 (507h58m41.179810515s) [seg_pos={0 0}, upload=445758]: Done begin upload segment.
Dec 26 19:27:41.308777845 (507h58m41.17986088s) [noise=true , address=77.90.31.21:4877     , piece_id=O2P4YQO6LWZCB7AGAEQZWHUGD5KFGSMFLGQ6Q45EEEDHTBCYGXLA, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.308784939 (507h58m41.179867984s) [noise=true , address=45.91.116.7:4012     , piece_id=2HTHDBMFBNK57TG7HLF5WSV3SOPLZPAAYN4FURH4RSQ2TGEPTJWA, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.309341018 (507h58m41.180424063s) [noise=true , address=77.90.31.14:4663     , piece_id=6R3DSFZ7R5DNWU2WLKMXIYFQLKLDAO25L5UUSFNNULNIA3G2C5DQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.309500321 (507h58m41.180583356s) [noise=true , address=45.91.116.14:4189    , piece_id=FYPN5P3UX26IN3WWH32H7EKIY573XOXNYVPEC66G233C6WGESKQQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.309897037 (507h58m41.180980072s) [noise=true , address=77.90.31.10:4520     , piece_id=UJQNPXGGVTJM4BV35NCEVOUNPKLEEK4HXSZDSBVWSALQWZ2DX4SQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.309939598 (507h58m41.181022633s) [noise=true , address=145.79.227.49:4339   , piece_id=5S3X2XLFEP6ZCTFNTW2TS4RRP2K6QEKYIGKVJQJJWIHUPCTTXXKQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.310456903 (507h58m41.181539948s) [noise=true , address=45.91.116.5:4297     , piece_id=ZWU2RXIHKPM7WC2EXRCGCAJGGY25VOLWZ6XBER7C25V76WTL7JRQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.311146315 (507h58m41.18222936s) [noise=true , address=45.91.116.13:4156    , piece_id=YFMC5JXF4NE7FXLLPLZTPM37XC3PRPIBHCN23UCNSYUP62BRWDHQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.311898197 (507h58m41.182981232s) [noise=true , address=45.91.116.6:4976     , piece_id=3QSWQQUVFPENRSPFFLT67JISMOWMVBWUPUBEPKUVKEUWDIV6TDZA, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.312501115 (507h58m41.18358415s) [noise=true , address=145.79.227.46:4601   , piece_id=M5Q3IEZNDGYULCWTSIZC64JDMD7O5A2BTB6B7ZJHKWLIYGNLNMEQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.313163576 (507h58m41.184246611s) [noise=true , address=145.79.227.45:4229   , piece_id=BAGK7P6P4RE6IYZRSR3CUBFMM2735KFOKTSDECYRIUVOD4WMFOIQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.315011475 (507h58m41.18609452s) [noise=true , address=145.79.227.41:4085   , piece_id=6H6OPJTA36IGAA44MEB7TRUM57FXLHUTKAHAFVO7TUYCHSZUFZDQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.320588425 (507h58m41.19167147s) [noise=true , address=145.79.227.39:4013   , piece_id=4RBG7SCJ4LPWC7Z7WVBCSZSREMKEGXGF24LQWIF4YTLY4NXLQ4YA, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.322580969 (507h58m41.193664004s) [noise=true , address=145.79.227.38:4978   , piece_id=K54VTN3PCPRU25YRPZY6BMOKLNPXAI7KLTXJZ5A3FU5G2TSPY73Q, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.325330164 (507h58m41.196413209s) [noise=true , address=145.79.227.31:4799   , piece_id=HFLZ3NWOGQUGW652C2K4F7AO62JO33WPECV6XOMEAGMMLIFNL3VA, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:41.329506136 (507h58m41.200589181s) [noise=true , address=145.79.227.52:4445   , piece_id=GEY5C2RN52JCFMQYFRJNYZUWOFR3B2OMSAXLFJNYRHALWOM2NEIQ, seg_pos={0 0}, upload=445758]: Uploading piece...
Dec 26 19:27:43.036143702 (507h58m42.907226747s) [upload=445758]: Next returned nil segment
Dec 26 19:27:43.036155795 (507h58m42.90723884s) [upload=445758]: Waiting for error group managing segments...
Dec 26 19:27:43.311910360 (507h58m43.182993395s) [noise=true , address=145.79.227.49:4339   , piece_id=5S3X2XLFEP6ZCTFNTW2TS4RRP2K6QEKYIGKVJQJJWIHUPCTTXXKQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.327633199 (507h58m43.198716234s) [noise=true , address=145.79.227.39:4013   , piece_id=4RBG7SCJ4LPWC7Z7WVBCSZSREMKEGXGF24LQWIF4YTLY4NXLQ4YA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.329777191 (507h58m43.200860226s) [noise=true , address=145.79.227.31:4799   , piece_id=HFLZ3NWOGQUGW652C2K4F7AO62JO33WPECV6XOMEAGMMLIFNL3VA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.340871687 (507h58m43.211954722s) [noise=true , address=145.79.227.38:4978   , piece_id=K54VTN3PCPRU25YRPZY6BMOKLNPXAI7KLTXJZ5A3FU5G2TSPY73Q, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.341378092 (507h58m43.212461127s) [noise=true , address=145.79.227.45:4229   , piece_id=BAGK7P6P4RE6IYZRSR3CUBFMM2735KFOKTSDECYRIUVOD4WMFOIQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.347898938 (507h58m43.218981973s) [noise=true , address=145.79.227.52:4445   , piece_id=GEY5C2RN52JCFMQYFRJNYZUWOFR3B2OMSAXLFJNYRHALWOM2NEIQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.357660586 (507h58m43.228743621s) [noise=true , address=145.79.227.46:4601   , piece_id=M5Q3IEZNDGYULCWTSIZC64JDMD7O5A2BTB6B7ZJHKWLIYGNLNMEQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.419216029 (507h58m43.290299064s) [noise=true , address=145.79.227.41:4085   , piece_id=6H6OPJTA36IGAA44MEB7TRUM57FXLHUTKAHAFVO7TUYCHSZUFZDQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.503482356 (507h58m43.374565402s) [noise=true , address=77.90.31.14:4663     , piece_id=6R3DSFZ7R5DNWU2WLKMXIYFQLKLDAO25L5UUSFNNULNIA3G2C5DQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.554993453 (507h58m43.426076488s) [noise=true , address=45.91.116.13:4156    , piece_id=YFMC5JXF4NE7FXLLPLZTPM37XC3PRPIBHCN23UCNSYUP62BRWDHQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.581941526 (507h58m43.453024561s) [noise=true , address=77.90.31.21:4877     , piece_id=O2P4YQO6LWZCB7AGAEQZWHUGD5KFGSMFLGQ6Q45EEEDHTBCYGXLA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.642103044 (507h58m43.513186079s) [noise=true , address=45.91.116.5:4297     , piece_id=ZWU2RXIHKPM7WC2EXRCGCAJGGY25VOLWZ6XBER7C25V76WTL7JRQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.714436521 (507h58m43.585519556s) [noise=true , address=45.91.116.6:4976     , piece_id=3QSWQQUVFPENRSPFFLT67JISMOWMVBWUPUBEPKUVKEUWDIV6TDZA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.714459804 (507h58m43.585542859s) [seg_pos={0 0}, upload=445758]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:27:43.714815792 (507h58m43.585898827s) [noise=true , address=45.91.116.14:4189    , piece_id=FYPN5P3UX26IN3WWH32H7EKIY573XOXNYVPEC66G233C6WGESKQQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 26 19:27:43.714843725 (507h58m43.58592677s) [noise=true , address=45.91.116.14:4189    , piece_id=FYPN5P3UX26IN3WWH32H7EKIY573XOXNYVPEC66G233C6WGESKQQ, seg_pos={0 0}, upload=445758]: Long tail cancellation due to optimal threshold
Dec 26 19:27:43.715168605 (507h58m43.58625164s) [noise=true , address=45.91.116.7:4012     , piece_id=2HTHDBMFBNK57TG7HLF5WSV3SOPLZPAAYN4FURH4RSQ2TGEPTJWA, seg_pos={0 0}, upload=445758]: Done uploading piece. err: <nil>
Dec 26 19:27:43.715246663 (507h58m43.586329718s) [noise=true , address=77.90.31.10:4520     , piece_id=UJQNPXGGVTJM4BV35NCEVOUNPKLEEK4HXSZDSBVWSALQWZ2DX4SQ, seg_pos={0 0}, upload=445758]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 26 19:27:43.715255109 (507h58m43.586338144s) [noise=true , address=77.90.31.10:4520     , piece_id=UJQNPXGGVTJM4BV35NCEVOUNPKLEEK4HXSZDSBVWSALQWZ2DX4SQ, seg_pos={0 0}, upload=445758]: Long tail cancellation due to optimal threshold
Dec 26 19:27:43.715274667 (507h58m43.586357702s) [seg_pos={0 0}, upload=445758]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 26 19:27:43.715349048 (507h58m43.586432083s) [upload=445758]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:27:43.715355821 (507h58m43.586438856s) [upload=445758]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:27:44.015517451 (507h58m43.886600496s) [upload=445758]: Done uploading segments...
Dec 26 19:27:44.015545825 (507h58m43.88662886s) [upload=445758]: Upload finished. err: <nil>
Duration: 30.0017577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.X]
duration: 
30.002821637s
trace: 
https://jaeger.dev.storj.io/trace/5cf14e31711fa112
FAIL (2025-12-26 19:27:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
07h57m46.181626598s) [upload=445745]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:26:46.558397721 (507h57m46.429480756s) [seg_pos={0 0}, upload=445745]: Begin upload segment...
Dec 26 19:26:46.558461483 (507h57m46.429544528s) [seg_pos={0 0}, upload=445745]: Done begin upload segment.
Dec 26 19:26:46.558512139 (507h57m46.429595174s) [noise=true , address=145.79.227.31:4799   , piece_id=D23KY7Z3EWTZRBQVRCGKSROEJU5FHQOQK3YOTJRMH2G73AYHVAMA, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.558584446 (507h57m46.429667491s) [noise=true , address=145.79.227.19:4078   , piece_id=DSMGTXQHOOWHQZK5TMIKCRRVURMRZGW4ITQLWGCI5VPJZJYGDQ4A, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.559344594 (507h57m46.430427629s) [noise=true , address=145.79.227.37:4566   , piece_id=TO7ZWTW5D7RU4M74USQNIEKL6CK4WKVXNZX5K6WJXIQBOSHXEEGA, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.559836811 (507h57m46.430919846s) [noise=true , address=145.79.227.5:4221    , piece_id=F7ZH3PGVVDT6UFOMNCYSBGXRTMRHWBNFYDA4JSXJLNNCJ2YX3OCQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.560341412 (507h57m46.431424477s) [noise=true , address=145.79.227.38:4977   , piece_id=PPIG3XW3TXUVVBIS5J65IZ4WM3ASAB4J34XLZ53YPWVBKQQFXKRQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.560862735 (507h57m46.43194578s) [noise=true , address=45.91.116.14:4190    , piece_id=2K3353VUW2I2KBOCLLLWDGMERHFNLDSSEBNFR7EGY52VOIHLWDZQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.561407251 (507h57m46.432490296s) [noise=true , address=45.91.116.12:4227    , piece_id=FQ7KM3GU45ETPFADASPRBXHV535AIFPSQQJSALMMELX5WNO5ZMCQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.561934767 (507h57m46.433017812s) [noise=true , address=145.79.227.49:4338   , piece_id=X3JQGLVGVDUYYHPWERSGBGJK74PH2KSLLVENV5ODWTGFJ6LW4IOA, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.562337653 (507h57m46.433420688s) [noise=true , address=145.79.227.6:4257    , piece_id=EYHP7CQELXFTSYYEG6T6GVHX7WAXI325YN7BFQ3C4CWTBXCTZQVQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.562904202 (507h57m46.433987237s) [noise=true , address=145.79.227.50:4374   , piece_id=54QIJJQDG4XS3NHCBWSHLQVF33KFG3B7OZF2LRXFPLQGGOVR7SYA, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.564251387 (507h57m46.435334432s) [noise=true , address=77.90.31.17:4773     , piece_id=YA5COEY7NLE77SXXZGVTSA32BQQYDDH7IAMSC6EXEP6V2B4L74FQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.565609073 (507h57m46.436692108s) [noise=true , address=145.79.227.39:4015   , piece_id=SJEJOLCVIXHD6KQ2NF3QWWT5YJEP4AMCMCW2BEXRAJ6XD4WHHXIA, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.568821780 (507h57m46.439904815s) [noise=true , address=145.79.227.3:4151    , piece_id=BKD7F2XBZC5SYGOX2NVTS6UITKJ24MOQ6Z2PZHWBRDB7IFYZO3QQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.572671070 (507h57m46.443754125s) [noise=true , address=45.91.116.11:4118    , piece_id=QWEXSHDSNKB2JFCWH2X3ZOAO3LMPIN3EXFO4NPQGEC45E7TDRXLA, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.575938442 (507h57m46.447021477s) [noise=true , address=145.79.227.52:4445   , piece_id=TWEHRAHXCZ5T6SM7PH2RTPUHKQNT5O27C4AVLUP6US6VQBTYBVBQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:46.577463897 (507h57m46.448546942s) [noise=true , address=45.91.116.8:4045     , piece_id=XNFUL4XOEPIAJSHUL6ZSTEHOUJGBSGM5LRCV6JGCTWUCU72U7NTQ, seg_pos={0 0}, upload=445745]: Uploading piece...
Dec 26 19:26:48.278009493 (507h57m48.149092528s) [upload=445745]: Next returned nil segment
Dec 26 19:26:48.278029751 (507h57m48.149112796s) [upload=445745]: Waiting for error group managing segments...
Dec 26 19:26:48.565447053 (507h57m48.436530088s) [noise=true , address=145.79.227.31:4799   , piece_id=D23KY7Z3EWTZRBQVRCGKSROEJU5FHQOQK3YOTJRMH2G73AYHVAMA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.568483436 (507h57m48.439566481s) [noise=true , address=145.79.227.38:4977   , piece_id=PPIG3XW3TXUVVBIS5J65IZ4WM3ASAB4J34XLZ53YPWVBKQQFXKRQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.578588057 (507h57m48.449671102s) [noise=true , address=145.79.227.37:4566   , piece_id=TO7ZWTW5D7RU4M74USQNIEKL6CK4WKVXNZX5K6WJXIQBOSHXEEGA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.578873851 (507h57m48.449956886s) [noise=true , address=145.79.227.52:4445   , piece_id=TWEHRAHXCZ5T6SM7PH2RTPUHKQNT5O27C4AVLUP6US6VQBTYBVBQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.584278612 (507h57m48.455361647s) [noise=true , address=145.79.227.50:4374   , piece_id=54QIJJQDG4XS3NHCBWSHLQVF33KFG3B7OZF2LRXFPLQGGOVR7SYA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.595013924 (507h57m48.46609697s) [noise=true , address=145.79.227.49:4338   , piece_id=X3JQGLVGVDUYYHPWERSGBGJK74PH2KSLLVENV5ODWTGFJ6LW4IOA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.598795354 (507h57m48.469878399s) [noise=true , address=145.79.227.39:4015   , piece_id=SJEJOLCVIXHD6KQ2NF3QWWT5YJEP4AMCMCW2BEXRAJ6XD4WHHXIA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.608734831 (507h57m48.479817866s) [noise=true , address=145.79.227.3:4151    , piece_id=BKD7F2XBZC5SYGOX2NVTS6UITKJ24MOQ6Z2PZHWBRDB7IFYZO3QQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.790524801 (507h57m48.661607846s) [noise=true , address=45.91.116.12:4227    , piece_id=FQ7KM3GU45ETPFADASPRBXHV535AIFPSQQJSALMMELX5WNO5ZMCQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.791413293 (507h57m48.662496328s) [noise=true , address=77.90.31.17:4773     , piece_id=YA5COEY7NLE77SXXZGVTSA32BQQYDDH7IAMSC6EXEP6V2B4L74FQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.850543698 (507h57m48.721626743s) [noise=true , address=45.91.116.14:4190    , piece_id=2K3353VUW2I2KBOCLLLWDGMERHFNLDSSEBNFR7EGY52VOIHLWDZQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:48.926229337 (507h57m48.797312382s) [noise=true , address=45.91.116.8:4045     , piece_id=XNFUL4XOEPIAJSHUL6ZSTEHOUJGBSGM5LRCV6JGCTWUCU72U7NTQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:49.689915969 (507h57m49.560999014s) [noise=true , address=45.91.116.11:4118    , piece_id=QWEXSHDSNKB2JFCWH2X3ZOAO3LMPIN3EXFO4NPQGEC45E7TDRXLA, seg_pos={0 0}, upload=445745]: Done uploading piece. err: <nil>
Dec 26 19:26:49.689942380 (507h57m49.561025415s) [seg_pos={0 0}, upload=445745]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:26:49.690159282 (507h57m49.561242317s) [noise=true , address=145.79.227.19:4078   , piece_id=DSMGTXQHOOWHQZK5TMIKCRRVURMRZGW4ITQLWGCI5VPJZJYGDQ4A, seg_pos={0 0}, upload=445745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:59256->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:59256->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59256->145.79.227.19:4078: use of closed network connection
Dec 26 19:26:49.690167197 (507h57m49.561250252s) [noise=true , address=145.79.227.19:4078   , piece_id=DSMGTXQHOOWHQZK5TMIKCRRVURMRZGW4ITQLWGCI5VPJZJYGDQ4A, seg_pos={0 0}, upload=445745]: Long tail cancellation due to optimal threshold
Dec 26 19:26:49.690198667 (507h57m49.561281702s) [noise=true , address=145.79.227.6:4257    , piece_id=EYHP7CQELXFTSYYEG6T6GVHX7WAXI325YN7BFQ3C4CWTBXCTZQVQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:46082->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:46082->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46082->145.79.227.6:4257: use of closed network connection
Dec 26 19:26:49.690209038 (507h57m49.561292073s) [noise=true , address=145.79.227.6:4257    , piece_id=EYHP7CQELXFTSYYEG6T6GVHX7WAXI325YN7BFQ3C4CWTBXCTZQVQ, seg_pos={0 0}, upload=445745]: Long tail cancellation due to optimal threshold
Dec 26 19:26:49.690348513 (507h57m49.561431558s) [noise=true , address=145.79.227.5:4221    , piece_id=F7ZH3PGVVDT6UFOMNCYSBGXRTMRHWBNFYDA4JSXJLNNCJ2YX3OCQ, seg_pos={0 0}, upload=445745]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:49458->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:49458->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->145.79.227.5:4221: use of closed network connection
Dec 26 19:26:49.690357179 (507h57m49.561440225s) [noise=true , address=145.79.227.5:4221    , piece_id=F7ZH3PGVVDT6UFOMNCYSBGXRTMRHWBNFYDA4JSXJLNNCJ2YX3OCQ, seg_pos={0 0}, upload=445745]: Long tail cancellation due to optimal threshold
Dec 26 19:26:49.690377147 (507h57m49.561460192s) [seg_pos={0 0}, upload=445745]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:26:49.690473090 (507h57m49.561556135s) [upload=445745]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:26:49.690479582 (507h57m49.561562617s) [upload=445745]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:26:49.990591157 (507h57m49.861674212s) [upload=445745]: Done uploading segments...
Dec 26 19:26:49.990616936 (507h57m49.861699971s) [upload=445745]: Upload finished. err: <nil>
Duration: 30.00290669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X.]
duration: 
6.145797684s
trace: 
https://jaeger.dev.storj.io/trace/360f40575f4edd2b
FAIL (2025-12-26 19:25:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h56m1.278588598s) [upload=445722]: Uploading segments...
Dec 26 19:25:01.407644558 (507h56m1.278727603s) [upload=445722]: Got next segment. Inline: false
Dec 26 19:25:01.407693922 (507h56m1.278776967s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:25:01.407700344 (507h56m1.278783389s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:25:01.657179045 (507h56m1.52826209s) [seg_pos={0 0}, upload=445722]: Begin upload segment...
Dec 26 19:25:01.657242936 (507h56m1.528325981s) [seg_pos={0 0}, upload=445722]: Done begin upload segment.
Dec 26 19:25:01.657299795 (507h56m1.52838284s) [noise=true , address=145.79.227.47:4267   , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.657308621 (507h56m1.528391666s) [noise=true , address=45.91.116.11:4119    , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.657990390 (507h56m1.529073435s) [noise=true , address=145.79.227.15:4873   , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.658037429 (507h56m1.529120504s) [noise=true , address=145.79.227.37:4565   , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.658388147 (507h56m1.529471182s) [noise=true , address=145.79.227.46:4603   , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.658746029 (507h56m1.529829104s) [noise=true , address=77.90.31.10:4520     , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.659188842 (507h56m1.530271887s) [noise=true , address=145.79.227.6:4257    , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.659525764 (507h56m1.530608799s) [noise=true , address=77.90.31.12:4590     , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.659840793 (507h56m1.530923838s) [noise=true , address=145.79.227.4:4185    , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.660203183 (507h56m1.531286228s) [noise=true , address=145.79.227.42:4121   , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.660577847 (507h56m1.531660892s) [noise=true , address=45.91.116.8:4045     , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.660938073 (507h56m1.532021118s) [noise=true , address=145.79.227.51:4410   , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.661266889 (507h56m1.532349934s) [noise=true , address=145.79.227.38:4978   , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.661551861 (507h56m1.532634906s) [noise=true , address=45.91.116.7:4011     , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.662362496 (507h56m1.533445531s) [noise=true , address=145.79.227.8:4330    , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.663593539 (507h56m1.534676584s) [noise=true , address=77.90.31.18:4808     , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:03.289076833 (507h56m3.160159878s) [upload=445722]: Next returned nil segment
Dec 26 19:25:03.289095970 (507h56m3.160179005s) [upload=445722]: Waiting for error group managing segments...
Dec 26 19:25:03.685430911 (507h56m3.556513956s) [noise=true , address=145.79.227.38:4978   , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.687006622 (507h56m3.558089667s) [noise=true , address=145.79.227.8:4330    , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.691721880 (507h56m3.562804925s) [noise=true , address=145.79.227.37:4565   , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.692105290 (507h56m3.563188335s) [noise=true , address=145.79.227.46:4603   , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.692772531 (507h56m3.563855576s) [noise=true , address=145.79.227.47:4267   , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.696193235 (507h56m3.56727629s) [noise=true , address=145.79.227.42:4121   , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.698467325 (507h56m3.56955037s) [noise=true , address=145.79.227.51:4410   , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.731878005 (507h56m3.60296105s) [noise=true , address=145.79.227.15:4873   , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.855972482 (507h56m3.727055528s) [noise=true , address=77.90.31.18:4808     , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.942103340 (507h56m3.813186385s) [noise=true , address=45.91.116.7:4011     , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.950017160 (507h56m3.821100205s) [noise=true , address=45.91.116.11:4119    , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:04.053891054 (507h56m3.924974099s) [noise=true , address=45.91.116.8:4045     , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:04.085425060 (507h56m3.956508105s) [noise=true , address=77.90.31.10:4520     , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:04.085451440 (507h56m3.956534485s) [seg_pos={0 0}, upload=445722]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:25:04.085663595 (507h56m3.95674664s) [noise=true , address=145.79.227.4:4185    , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection
Dec 26 19:25:04.085670157 (507h56m3.956753202s) [noise=true , address=145.79.227.4:4185    , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold
Dec 26 19:25:04.085854888 (507h56m3.956937923s) [noise=true , address=77.90.31.12:4590     , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:25:04.085873264 (507h56m3.956956299s) [noise=true , address=77.90.31.12:4590     , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold
Dec 26 19:25:04.086049079 (507h56m3.957132144s) [noise=true , address=145.79.227.6:4257    , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection
Dec 26 19:25:04.086060330 (507h56m3.957143375s) [noise=true , address=145.79.227.6:4257    , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold
Dec 26 19:25:04.086087442 (507h56m3.957170487s) [seg_pos={0 0}, upload=445722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:25:04.086189787 (507h56m3.957272832s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:25:04.086195668 (507h56m3.957278713s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:25:04.388933212 (507h56m4.260016268s) [upload=445722]: Done uploading segments...
Dec 26 19:25:04.388956417 (507h56m4.260039462s) [upload=445722]: Upload finished. err: <nil>
Duration: 6.145815126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....X]
duration: 
30.002326734s
trace: 
https://jaeger.dev.storj.io/trace/360f40575f4edd2b
FAIL (2025-12-26 19:25:30 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h56m1.278588598s) [upload=445722]: Uploading segments...
Dec 26 19:25:01.407644558 (507h56m1.278727603s) [upload=445722]: Got next segment. Inline: false
Dec 26 19:25:01.407693922 (507h56m1.278776967s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:25:01.407700344 (507h56m1.278783389s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:25:01.657179045 (507h56m1.52826209s) [seg_pos={0 0}, upload=445722]: Begin upload segment...
Dec 26 19:25:01.657242936 (507h56m1.528325981s) [seg_pos={0 0}, upload=445722]: Done begin upload segment.
Dec 26 19:25:01.657299795 (507h56m1.52838284s) [noise=true , address=145.79.227.47:4267   , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.657308621 (507h56m1.528391666s) [noise=true , address=45.91.116.11:4119    , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.657990390 (507h56m1.529073435s) [noise=true , address=145.79.227.15:4873   , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.658037429 (507h56m1.529120504s) [noise=true , address=145.79.227.37:4565   , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.658388147 (507h56m1.529471182s) [noise=true , address=145.79.227.46:4603   , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.658746029 (507h56m1.529829104s) [noise=true , address=77.90.31.10:4520     , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.659188842 (507h56m1.530271887s) [noise=true , address=145.79.227.6:4257    , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.659525764 (507h56m1.530608799s) [noise=true , address=77.90.31.12:4590     , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.659840793 (507h56m1.530923838s) [noise=true , address=145.79.227.4:4185    , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.660203183 (507h56m1.531286228s) [noise=true , address=145.79.227.42:4121   , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.660577847 (507h56m1.531660892s) [noise=true , address=45.91.116.8:4045     , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.660938073 (507h56m1.532021118s) [noise=true , address=145.79.227.51:4410   , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.661266889 (507h56m1.532349934s) [noise=true , address=145.79.227.38:4978   , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.661551861 (507h56m1.532634906s) [noise=true , address=45.91.116.7:4011     , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.662362496 (507h56m1.533445531s) [noise=true , address=145.79.227.8:4330    , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:01.663593539 (507h56m1.534676584s) [noise=true , address=77.90.31.18:4808     , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Uploading piece...
Dec 26 19:25:03.289076833 (507h56m3.160159878s) [upload=445722]: Next returned nil segment
Dec 26 19:25:03.289095970 (507h56m3.160179005s) [upload=445722]: Waiting for error group managing segments...
Dec 26 19:25:03.685430911 (507h56m3.556513956s) [noise=true , address=145.79.227.38:4978   , piece_id=PZOZALTDLSEFO2SJNP54TVCI4JHBAVVQI4NKR7VN2BKSZBPEL62Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.687006622 (507h56m3.558089667s) [noise=true , address=145.79.227.8:4330    , piece_id=U4LFLTQDT7SGRO5IIP3PCLSSWNA42CUPKPDDYFAZP4UAX5PTOG2A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.691721880 (507h56m3.562804925s) [noise=true , address=145.79.227.37:4565   , piece_id=6AHTF6RQ76A2MR2IAG7V6AJ5MYZPPR367327D6NHB3NJ26SGWE6Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.692105290 (507h56m3.563188335s) [noise=true , address=145.79.227.46:4603   , piece_id=2HOS7RDUD73RVXYS3Z2SNWR5RH6YD5AN5VSDWI2R6JACNJCGFFBA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.692772531 (507h56m3.563855576s) [noise=true , address=145.79.227.47:4267   , piece_id=FETXS2MBOLWO5CUY7BOIDV4VAEEKVXZPHRW3AO56UUVNLEPIP67A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.696193235 (507h56m3.56727629s) [noise=true , address=145.79.227.42:4121   , piece_id=PXOPHAFX7HVIZAAG7IJLCBMH47KHNOA7TWW3T23Q4N2AYTTAPB2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.698467325 (507h56m3.56955037s) [noise=true , address=145.79.227.51:4410   , piece_id=DNDEWSY5AW5BHM4M6Z6E4ODHA6WSNGHIA5GNQKYPYL6VVYVZXYGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.731878005 (507h56m3.60296105s) [noise=true , address=145.79.227.15:4873   , piece_id=OOXL562IEKFINV33FNW3264QF6NEETQWLM74YOPIRAKKL6UGLNOA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.855972482 (507h56m3.727055528s) [noise=true , address=77.90.31.18:4808     , piece_id=UV7CAJIXLA5LJWSN4LKSP2376KY3CV6DX7I7MXJT4NPB26LBOBGA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.942103340 (507h56m3.813186385s) [noise=true , address=45.91.116.7:4011     , piece_id=WW65UELB7XJGUWMW5EZXPSY7GPVVGJMMVHUJT55V5BNBMU5KTU4A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:03.950017160 (507h56m3.821100205s) [noise=true , address=45.91.116.11:4119    , piece_id=ZUNSEPXAHQXOV7BIGDSWZ6GGEBCGREMCRICWV2NVMPODXK7UUFHA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:04.053891054 (507h56m3.924974099s) [noise=true , address=45.91.116.8:4045     , piece_id=FHKV65OZ4R7FMFVKAVYPVLEHZLQRIYHUXC7AGMLQCSPAQWTZRI2Q, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:04.085425060 (507h56m3.956508105s) [noise=true , address=77.90.31.10:4520     , piece_id=BSQDSDIMCDH7OHIAS37ZNL6GJ4AQ6RE25C2ANYIXDDO5WCTZVYEA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: <nil>
Dec 26 19:25:04.085451440 (507h56m3.956534485s) [seg_pos={0 0}, upload=445722]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:25:04.085663595 (507h56m3.95674664s) [noise=true , address=145.79.227.4:4185    , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53482->145.79.227.4:4185: use of closed network connection
Dec 26 19:25:04.085670157 (507h56m3.956753202s) [noise=true , address=145.79.227.4:4185    , piece_id=QWXAWDFZJYFFBBCK2FN23G4PJSHX77I3SDEOAIFQG5SQMOEBJEGQ, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold
Dec 26 19:25:04.085854888 (507h56m3.956937923s) [noise=true , address=77.90.31.12:4590     , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:25:04.085873264 (507h56m3.956956299s) [noise=true , address=77.90.31.12:4590     , piece_id=4F6R2GWLMDEMDWKTKJMHACU53RSHRESJ2BCQ3VZZKRXW55B5XHSA, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold
Dec 26 19:25:04.086049079 (507h56m3.957132144s) [noise=true , address=145.79.227.6:4257    , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41330->145.79.227.6:4257: use of closed network connection
Dec 26 19:25:04.086060330 (507h56m3.957143375s) [noise=true , address=145.79.227.6:4257    , piece_id=K4LC4YBB5MMJ77WPEMNB6VORHYC44TF5R3UIM7CY2FCNL7LERB5A, seg_pos={0 0}, upload=445722]: Long tail cancellation due to optimal threshold
Dec 26 19:25:04.086087442 (507h56m3.957170487s) [seg_pos={0 0}, upload=445722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:25:04.086189787 (507h56m3.957272832s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:25:04.086195668 (507h56m3.957278713s) [upload=445722]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:25:04.388933212 (507h56m4.260016268s) [upload=445722]: Done uploading segments...
Dec 26 19:25:04.388956417 (507h56m4.260039462s) [upload=445722]: Upload finished. err: <nil>
Duration: 30.002350218s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [XX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X....]
duration: 
6.097242214s
trace: 
https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19
FAIL (2025-12-26 19:24:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false
Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment...
Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment.
Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment
Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments...
Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection
Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection
Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments...
Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil>
Duration: 6.097268906s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X...]
duration: 
10.602774485s
trace: 
https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19
FAIL (2025-12-26 19:24:19 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false
Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment...
Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment.
Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment
Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments...
Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection
Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection
Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments...
Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil>
Duration: 10.602811286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X..]
duration: 
6.150134131s
trace: 
https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19
FAIL (2025-12-26 19:23:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false
Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment...
Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment.
Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment
Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments...
Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection
Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection
Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments...
Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil>
Duration: 6.150166372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [..XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X.]
duration: 
6.983247294s
trace: 
https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19
FAIL (2025-12-26 19:22:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false
Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment...
Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment.
Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment
Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments...
Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection
Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection
Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments...
Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil>
Duration: 6.983279655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [...XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.X]
duration: 
30.003050232s
trace: 
https://jaeger.dev.storj.io/trace/6b1efe220e6cdc19
FAIL (2025-12-26 19:22:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ments...
Dec 26 19:21:34.147184597 (507h52m34.018267632s) [upload=445672]: Got next segment. Inline: false
Dec 26 19:21:34.147223891 (507h52m34.018306926s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:21:34.147229722 (507h52m34.018312757s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:21:34.404276014 (507h52m34.275359059s) [seg_pos={0 0}, upload=445672]: Begin upload segment...
Dec 26 19:21:34.404328804 (507h52m34.275411839s) [seg_pos={0 0}, upload=445672]: Done begin upload segment.
Dec 26 19:21:34.404357939 (507h52m34.275440974s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.404424095 (507h52m34.27550714s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405356872 (507h52m34.276439907s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405400274 (507h52m34.276483309s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.405826325 (507h52m34.27690936s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406125666 (507h52m34.277208701s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406425596 (507h52m34.277508631s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406471775 (507h52m34.27755481s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.406884942 (507h52m34.277967977s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407065014 (507h52m34.278148049s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407156979 (507h52m34.278240024s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407458363 (507h52m34.278541398s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.407721775 (507h52m34.27880481s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408473826 (507h52m34.279556861s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.408900899 (507h52m34.279983934s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:34.409255836 (507h52m34.280338971s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Uploading piece...
Dec 26 19:21:36.116453418 (507h52m35.987536453s) [upload=445672]: Next returned nil segment
Dec 26 19:21:36.116471012 (507h52m35.987554057s) [upload=445672]: Waiting for error group managing segments...
Dec 26 19:21:36.397750706 (507h52m36.268833752s) [noise=true , address=145.79.227.45:4229   , piece_id=H6D65MVCSFIW6PE6TYNYETR4LULPB427W2QP23ZGFWMN2U7LVFOA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.406413793 (507h52m36.277496828s) [noise=true , address=145.79.227.38:4979   , piece_id=MPYW2QPOHJCU3Z33TH7LGXEXJXAGV4ZJQMGKIDTFCH3R4DKX4S5A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412071306 (507h52m36.283154341s) [noise=true , address=145.79.227.52:4447   , piece_id=WADNUPNA7NLN2KF6Z764Q5BNFISCVXR33XYLZISSQJ37TSHKQRLA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.412128274 (507h52m36.283211309s) [noise=true , address=145.79.227.37:4566   , piece_id=SG32DPJZ6DM5NBB6LMRZXFXNJNUOYOPXWCH3BIS47S6664EP6IZQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.414042890 (507h52m36.285125935s) [noise=true , address=145.79.227.50:4373   , piece_id=6XZO6LQOMITBWEN5YY52PRVTUJAE4KSZSA63JFY4GX3SE4QGRXTA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.416169199 (507h52m36.287252244s) [noise=true , address=145.79.227.21:4475   , piece_id=HQEZMT5IT7F6LDSWINXVQ5XBQ6BD64VSEHG54BYAN2544ILBRVIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.428731219 (507h52m36.299814254s) [noise=true , address=145.79.227.46:4601   , piece_id=ZF2HYUQGAUAHSN563RZCXJZD6XN37NZ7GRNNV5MVVAKJNKYRU4MA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.430456024 (507h52m36.301539059s) [noise=true , address=145.79.227.22:4509   , piece_id=OK36XRXQHKLUK3HQF6E7WPHZWSJIWP73SSLDTSZIS2WGQIRRMM3A, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.432583144 (507h52m36.303666179s) [noise=true , address=145.79.227.15:4873   , piece_id=YULFI2MY7I6MJUL6BOVZ6YRAT343XG3EBEZZKBG6LIYFYEPQYHKQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.441369835 (507h52m36.31245287s) [noise=true , address=145.79.227.39:4014   , piece_id=5JKWRXZDND3RVRAGQWRGCY7IZCI732RWTPBJ3WFU3YFJYV2YO7UQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.595495136 (507h52m36.466578182s) [noise=true , address=77.90.31.16:4736     , piece_id=WSUWPBXQSSHJVNRS4NXDUFOJIMGC74UBT3LT3HAJX2ODOZQUQYPA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.837896641 (507h52m36.708979676s) [noise=true , address=77.90.31.10:4519     , piece_id=DKSTQ4HPBOECGEQ24P5S5K7H6OA6VVWZ2SPDDDQZBJFB5MOHGIBA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919754113 (507h52m36.790837158s) [noise=true , address=45.91.116.6:4973     , piece_id=JZI4D7XOGHAWPAH4KG4TWUCPIC6SR7EJB7HOJAAVQ3PFXRGZFDQQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: <nil>
Dec 26 19:21:36.919772738 (507h52m36.790855783s) [seg_pos={0 0}, upload=445672]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:21:36.919939666 (507h52m36.791022711s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:21:36.919947160 (507h52m36.791030195s) [noise=true , address=145.79.227.41:4086   , piece_id=FBFG2UJPH3ODCBP377TMTRGDM2TYWVIU5AYKO5JTKIRCLEPXUKEA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920122453 (507h52m36.791205498s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44588->145.79.227.2:4115: use of closed network connection
Dec 26 19:21:36.920128265 (507h52m36.7912113s) [noise=true , address=145.79.227.2:4115    , piece_id=QIWTSOUWL4GRWBZRUOPFBZ4I64AYKOX5D54QHMDZVOVCKPPUC7HA, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920165386 (507h52m36.791248421s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52192->145.79.227.19:4078: use of closed network connection
Dec 26 19:21:36.920181336 (507h52m36.791264371s) [noise=true , address=145.79.227.19:4078   , piece_id=2SAEQXVIKI6UH3V6E6GIWJGINPN7J3E6A6PPU7ICKIWSX7BSGEIQ, seg_pos={0 0}, upload=445672]: Long tail cancellation due to optimal threshold
Dec 26 19:21:36.920233095 (507h52m36.79131613s) [seg_pos={0 0}, upload=445672]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:21:36.920318247 (507h52m36.791401292s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:21:36.920323106 (507h52m36.791406141s) [upload=445672]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:21:37.218908274 (507h52m37.089991309s) [upload=445672]: Done uploading segments...
Dec 26 19:21:37.218929785 (507h52m37.09001282s) [upload=445672]: Upload finished. err: <nil>
Duration: 30.003079167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX.]
duration: 
6.138537759s
trace: 
https://jaeger.dev.storj.io/trace/568ddfc5b1abce90
FAIL (2025-12-26 19:20:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ) context deadline exceeded; piecestore: (Node ID: 12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, Piece ID: INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ) context deadline exceeded
======LOGS======
ec 26 19:19:30.227211085 (507h50m30.09829413s) [upload=445644]: Starting upload
Dec 26 19:19:30.227295465 (507h50m30.09837852s) [upload=445644]: Done starting upload
Dec 26 19:19:30.227427417 (507h50m30.098510452s) [upload=445644]: Uploading segments...
Dec 26 19:19:30.227716587 (507h50m30.098799622s) [upload=445644]: Got next segment. Inline: false
Dec 26 19:19:30.227772574 (507h50m30.098855619s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:19:30.227778886 (507h50m30.098861941s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:19:30.476589825 (507h50m30.34767286s) [seg_pos={0 0}, upload=445644]: Begin upload segment...
Dec 26 19:19:30.476641673 (507h50m30.347724708s) [seg_pos={0 0}, upload=445644]: Done begin upload segment.
Dec 26 19:19:30.476666711 (507h50m30.347749746s) [noise=true , address=145.79.227.40:4050   , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.476722157 (507h50m30.347805192s) [noise=true , address=77.90.31.14:4663     , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477203974 (507h50m30.348287009s) [noise=true , address=145.79.227.36:4518   , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477215627 (507h50m30.348298662s) [noise=true , address=45.91.116.11:4119    , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477565563 (507h50m30.348648608s) [noise=true , address=77.90.31.10:4518     , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477769281 (507h50m30.348852316s) [noise=true , address=145.79.227.49:4337   , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477949394 (507h50m30.349032429s) [noise=true , address=45.91.116.9:4264     , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478139976 (507h50m30.349223011s) [noise=true , address=145.79.227.38:4977   , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478320461 (507h50m30.349403506s) [noise=true , address=145.79.227.11:4732   , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478486897 (507h50m30.349569932s) [noise=true , address=145.79.227.46:4601   , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478634619 (507h50m30.349717654s) [noise=true , address=77.90.31.16:4736     , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478860288 (507h50m30.349943333s) [noise=true , address=145.79.227.50:4373   , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478986719 (507h50m30.350069754s) [noise=true , address=45.91.116.13:4156    , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.479264238 (507h50m30.350347273s) [noise=true , address=145.79.227.45:4231   , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.479288273 (507h50m30.350371318s) [noise=true , address=145.79.227.48:4301   , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.479655703 (507h50m30.350738738s) [noise=true , address=145.79.227.15:4874   , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:32.181356449 (507h50m32.052439484s) [upload=445644]: Next returned nil segment
Dec 26 19:19:32.181390153 (507h50m32.052473198s) [upload=445644]: Waiting for error group managing segments...
Dec 26 19:19:32.474459137 (507h50m32.345542182s) [noise=true , address=145.79.227.50:4373   , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.479753769 (507h50m32.350836804s) [noise=true , address=145.79.227.38:4977   , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.485159292 (507h50m32.356242327s) [noise=true , address=145.79.227.48:4301   , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.487314757 (507h50m32.358397792s) [noise=true , address=145.79.227.40:4050   , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.490154434 (507h50m32.361237469s) [noise=true , address=145.79.227.11:4732   , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.493673595 (507h50m32.36475664s) [noise=true , address=145.79.227.46:4601   , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.495333936 (507h50m32.366416971s) [noise=true , address=145.79.227.36:4518   , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.496214773 (507h50m32.367297808s) [noise=true , address=145.79.227.15:4874   , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.497151808 (507h50m32.368234843s) [noise=true , address=145.79.227.45:4231   , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.507448064 (507h50m32.378531099s) [noise=true , address=145.79.227.49:4337   , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.729908207 (507h50m32.600991242s) [noise=true , address=45.91.116.13:4156    , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.814922039 (507h50m32.686005084s) [noise=true , address=45.91.116.9:4264     , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.880215529 (507h50m32.751298574s) [noise=true , address=77.90.31.14:4663     , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.880240045 (507h50m32.75132308s) [seg_pos={0 0}, upload=445644]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:19:32.880432512 (507h50m32.751515547s) [noise=true , address=77.90.31.10:4518     , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 26 19:19:32.880440587 (507h50m32.751523632s) [noise=true , address=77.90.31.10:4518     , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold
Dec 26 19:19:32.880449193 (507h50m32.751532238s) [noise=true , address=45.91.116.11:4119    , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:19:32.880457460 (507h50m32.751540495s) [noise=true , address=45.91.116.11:4119    , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold
Dec 26 19:19:32.880586004 (507h50m32.751669039s) [noise=true , address=77.90.31.16:4736     , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection
Dec 26 19:19:32.880592556 (507h50m32.751675601s) [noise=true , address=77.90.31.16:4736     , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold
Dec 26 19:19:32.880618526 (507h50m32.751701561s) [seg_pos={0 0}, upload=445644]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:19:32.880688279 (507h50m32.751771314s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:19:32.880694290 (507h50m32.751777325s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:19:33.182498829 (507h50m33.053581874s) [upload=445644]: Done uploading segments...
Dec 26 19:19:33.182520670 (507h50m33.053603705s) [upload=445644]: Upload finished. err: <nil>
Duration: 6.138565352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XXX]
duration: 
30.00262386s
trace: 
https://jaeger.dev.storj.io/trace/568ddfc5b1abce90
FAIL (2025-12-26 19:19:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ) context deadline exceeded; piecestore: (Node ID: 12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, Piece ID: INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ) context deadline exceeded
======LOGS======
ec 26 19:19:30.227211085 (507h50m30.09829413s) [upload=445644]: Starting upload
Dec 26 19:19:30.227295465 (507h50m30.09837852s) [upload=445644]: Done starting upload
Dec 26 19:19:30.227427417 (507h50m30.098510452s) [upload=445644]: Uploading segments...
Dec 26 19:19:30.227716587 (507h50m30.098799622s) [upload=445644]: Got next segment. Inline: false
Dec 26 19:19:30.227772574 (507h50m30.098855619s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:19:30.227778886 (507h50m30.098861941s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:19:30.476589825 (507h50m30.34767286s) [seg_pos={0 0}, upload=445644]: Begin upload segment...
Dec 26 19:19:30.476641673 (507h50m30.347724708s) [seg_pos={0 0}, upload=445644]: Done begin upload segment.
Dec 26 19:19:30.476666711 (507h50m30.347749746s) [noise=true , address=145.79.227.40:4050   , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.476722157 (507h50m30.347805192s) [noise=true , address=77.90.31.14:4663     , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477203974 (507h50m30.348287009s) [noise=true , address=145.79.227.36:4518   , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477215627 (507h50m30.348298662s) [noise=true , address=45.91.116.11:4119    , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477565563 (507h50m30.348648608s) [noise=true , address=77.90.31.10:4518     , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477769281 (507h50m30.348852316s) [noise=true , address=145.79.227.49:4337   , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.477949394 (507h50m30.349032429s) [noise=true , address=45.91.116.9:4264     , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478139976 (507h50m30.349223011s) [noise=true , address=145.79.227.38:4977   , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478320461 (507h50m30.349403506s) [noise=true , address=145.79.227.11:4732   , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478486897 (507h50m30.349569932s) [noise=true , address=145.79.227.46:4601   , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478634619 (507h50m30.349717654s) [noise=true , address=77.90.31.16:4736     , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478860288 (507h50m30.349943333s) [noise=true , address=145.79.227.50:4373   , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.478986719 (507h50m30.350069754s) [noise=true , address=45.91.116.13:4156    , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.479264238 (507h50m30.350347273s) [noise=true , address=145.79.227.45:4231   , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.479288273 (507h50m30.350371318s) [noise=true , address=145.79.227.48:4301   , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:30.479655703 (507h50m30.350738738s) [noise=true , address=145.79.227.15:4874   , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Uploading piece...
Dec 26 19:19:32.181356449 (507h50m32.052439484s) [upload=445644]: Next returned nil segment
Dec 26 19:19:32.181390153 (507h50m32.052473198s) [upload=445644]: Waiting for error group managing segments...
Dec 26 19:19:32.474459137 (507h50m32.345542182s) [noise=true , address=145.79.227.50:4373   , piece_id=STLHC2XIAWAJUF5B2QSR22LOTNMPL2DVPKV6IEOVCVVRM7OOCGTA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.479753769 (507h50m32.350836804s) [noise=true , address=145.79.227.38:4977   , piece_id=LWI4RXWMFJK2CROE7CSPOEVEB4TP2FKXPY3NWYLJDSJMLPXUWWFQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.485159292 (507h50m32.356242327s) [noise=true , address=145.79.227.48:4301   , piece_id=BJ2KPIU72PBVIPOXMJLPYLXQRQKIIBR74HUI7ZICP4BEYDD2L4XQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.487314757 (507h50m32.358397792s) [noise=true , address=145.79.227.40:4050   , piece_id=ZWXNCWZULTXE5LCG4QF7AQWWR34JCFTTZQVMJWEBDPHR3FJS2RUQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.490154434 (507h50m32.361237469s) [noise=true , address=145.79.227.11:4732   , piece_id=JYLU2BPQ255J4YVWS4XXPSE6FZBNQFBOT5EYE43AFSWPK7IFBHPQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.493673595 (507h50m32.36475664s) [noise=true , address=145.79.227.46:4601   , piece_id=XW4ED5LYEKC6CCETSJQXM2NTOTBGPGOTKZZKSCKFFEX43X3VZBDA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.495333936 (507h50m32.366416971s) [noise=true , address=145.79.227.36:4518   , piece_id=BXZPJQI3CAXRQEYGIMGSRYBTGAK4LU437WUHXOP6WIQ3NBQCSXYQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.496214773 (507h50m32.367297808s) [noise=true , address=145.79.227.15:4874   , piece_id=INW2KXB6L4Q5PT6Y4OXI55CHUKPGVCZDQVFXPW2SPEBVBPSENFQQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.497151808 (507h50m32.368234843s) [noise=true , address=145.79.227.45:4231   , piece_id=ECT3RFR36JKZ3OEDUIYAC3ZBC6N7TH5TBKZJKU4VNOLNZHNV7JFA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.507448064 (507h50m32.378531099s) [noise=true , address=145.79.227.49:4337   , piece_id=BB7ALN4BNYPJILNJ7FNXMC6ITAO7SMPP5R3HHMDRIQJTEFDGVR5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.729908207 (507h50m32.600991242s) [noise=true , address=45.91.116.13:4156    , piece_id=XKRT62G5LU7VLKQGV6M4WYS276BIKBIBBALUPKHEUOTFVN3SOBWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.814922039 (507h50m32.686005084s) [noise=true , address=45.91.116.9:4264     , piece_id=7DWE6EUWCORGIZMSQ4ZSI3ITDM5JXPGUHKRVVW4FAPYNBTEDADSQ, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.880215529 (507h50m32.751298574s) [noise=true , address=77.90.31.14:4663     , piece_id=FL6ENXYOQXN43CDISST24MCBT3F6HUMCPNL5ODU32K3VEC4NFEZA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: <nil>
Dec 26 19:19:32.880240045 (507h50m32.75132308s) [seg_pos={0 0}, upload=445644]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:19:32.880432512 (507h50m32.751515547s) [noise=true , address=77.90.31.10:4518     , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 26 19:19:32.880440587 (507h50m32.751523632s) [noise=true , address=77.90.31.10:4518     , piece_id=CZFBAYIQYA54TE22ZSZPQ2YKCW7JM6ND3HWH3LW25GNZES7JAG5A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold
Dec 26 19:19:32.880449193 (507h50m32.751532238s) [noise=true , address=45.91.116.11:4119    , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 26 19:19:32.880457460 (507h50m32.751540495s) [noise=true , address=45.91.116.11:4119    , piece_id=KGLFNOQAMT4SHOF2CTPRL7OB2MOLLU35CEVLBR74UQVTWZ53CDWA, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold
Dec 26 19:19:32.880586004 (507h50m32.751669039s) [noise=true , address=77.90.31.16:4736     , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44460->77.90.31.16:4736: use of closed network connection
Dec 26 19:19:32.880592556 (507h50m32.751675601s) [noise=true , address=77.90.31.16:4736     , piece_id=Z7K5MLYV7NORLAESOGY236TJMFOM7JH7O2UNFOKD6E6CIW6VVR2A, seg_pos={0 0}, upload=445644]: Long tail cancellation due to optimal threshold
Dec 26 19:19:32.880618526 (507h50m32.751701561s) [seg_pos={0 0}, upload=445644]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:19:32.880688279 (507h50m32.751771314s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:19:32.880694290 (507h50m32.751777325s) [upload=445644]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:19:33.182498829 (507h50m33.053581874s) [upload=445644]: Done uploading segments...
Dec 26 19:19:33.182520670 (507h50m33.053603705s) [upload=445644]: Upload finished. err: <nil>
Duration: 30.002641323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...XX]
duration: 
30.002375608s
trace: 
https://jaeger.dev.storj.io/trace/65ca9b941f274d1a
FAIL (2025-12-26 19:18:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
6 19:18:26.318350402 (507h49m26.189433437s) [upload=445627]: Starting upload
Dec 26 19:18:26.318422900 (507h49m26.189505935s) [upload=445627]: Done starting upload
Dec 26 19:18:26.318537800 (507h49m26.189620845s) [upload=445627]: Uploading segments...
Dec 26 19:18:26.318653029 (507h49m26.189736074s) [upload=445627]: Got next segment. Inline: false
Dec 26 19:18:26.318703836 (507h49m26.189786881s) [upload=445627]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:18:26.318710338 (507h49m26.189793373s) [upload=445627]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:18:26.572049420 (507h49m26.443132455s) [seg_pos={0 0}, upload=445627]: Begin upload segment...
Dec 26 19:18:26.572135163 (507h49m26.443218198s) [seg_pos={0 0}, upload=445627]: Done begin upload segment.
Dec 26 19:18:26.572203884 (507h49m26.443286929s) [noise=true , address=77.90.31.12:4590     , piece_id=VMELCTDYY7FCQQXJH2NPPRAZ64YEIY2B3PIVOES4AHPPSQ2TUAWA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.572237227 (507h49m26.443320262s) [noise=true , address=145.79.227.47:4266   , piece_id=SVGPB5X3YLU2UIKUGJS4GUNMLKL6BXZPJRQ3G66T6JZT6MI6BSZQ, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.572921571 (507h49m26.444004616s) [noise=true , address=145.79.227.52:4445   , piece_id=4KJEI3BIXC7EUXNDWDYFGU2NLHPKOJUWI77MCVYZDE6QP3NPLF3A, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.573405502 (507h49m26.444488537s) [noise=true , address=145.79.227.45:4230   , piece_id=7RUBFVPQDHIUQ5HUXGGH2BA3BPQWHGN2SMAANXCP3UO5EVYGVIEQ, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.573428435 (507h49m26.444511481s) [noise=true , address=145.79.227.50:4375   , piece_id=EZPMQA525ESJKSW6M365LF7GPW67RGM3NYWCJE3JQNVCQIRH433Q, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.573887591 (507h49m26.444970636s) [noise=true , address=145.79.227.30:4761   , piece_id=UWOPYRA6WTI75GRCAIKSYDPGZ7YEQPZAQP6ZM7ZIXPXQTDDE6YMA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.573911566 (507h49m26.444994611s) [noise=true , address=145.79.227.4:4185    , piece_id=Y7SJVXPFSLK6RZZWRTKSIMTKEVSBT2N5ZI57GUCVMOVX4XMH7FIA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.574211357 (507h49m26.445294392s) [noise=true , address=145.79.227.15:4875   , piece_id=EXHATTOQREEYUXA4ODVDSWQFDSYJUSPTTBT35DQYIARNSLFKDNFA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.574259498 (507h49m26.445342533s) [noise=true , address=145.79.227.49:4338   , piece_id=BQWZOZ3AOMVB7VOZ3C3TYEGIAFPCLEN64HEVSYV2RFS5EGH5A5PA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.574587613 (507h49m26.445670648s) [noise=true , address=45.91.116.5:4300     , piece_id=MWVUKWESPIC2T2UM5HAP3OQJQ6NOIOIZK6VCZ5TJHDADO63BCVCA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.574955584 (507h49m26.446038629s) [noise=true , address=145.79.227.39:4014   , piece_id=JN7CWFCIFS3IUFS7PWMVVMPCK7BOXLXHNVSNNVEFZN6CT3BDVKOA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.575284771 (507h49m26.446367806s) [noise=true , address=145.79.227.21:4475   , piece_id=5YKRLXVU7XSY3EC5PTQ4R3JLXAMOV2IIKAT65NYTLPJ4NUFVWTYA, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.575534045 (507h49m26.44661708s) [noise=true , address=45.91.116.14:4189    , piece_id=EVWJBUTXLTGWSTY3KMZOIEIYIL6H4KFDYLON26DKINZIPSQ7UA3Q, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.575582898 (507h49m26.446665933s) [noise=true , address=45.91.116.13:4156    , piece_id=PMTLLDWJ3SSDLD6JUW55M6MQ667LCXWJMWQYRDBQGA5ZIBZOTL5Q, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.575953935 (507h49m26.44703697s) [noise=true , address=77.90.31.14:4663     , piece_id=ENBFWB654DDFQFAS7CEYJUWBBCZW64OGGUZDRDCEP5MRNNDE7X3A, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:26.576256882 (507h49m26.447339917s) [noise=true , address=145.79.227.8:4330    , piece_id=GPZGJAV7FGQTP23ITFSSTIZWPC4ZMES6T74QZX452EIEYV4WTG7Q, seg_pos={0 0}, upload=445627]: Uploading piece...
Dec 26 19:18:28.285067696 (507h49m28.156150731s) [upload=445627]: Next returned nil segment
Dec 26 19:18:28.285079248 (507h49m28.156162283s) [upload=445627]: Waiting for error group managing segments...
Dec 26 19:18:28.575725339 (507h49m28.446808374s) [noise=true , address=145.79.227.49:4338   , piece_id=BQWZOZ3AOMVB7VOZ3C3TYEGIAFPCLEN64HEVSYV2RFS5EGH5A5PA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.577689810 (507h49m28.448772845s) [noise=true , address=145.79.227.39:4014   , piece_id=JN7CWFCIFS3IUFS7PWMVVMPCK7BOXLXHNVSNNVEFZN6CT3BDVKOA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.579691752 (507h49m28.450774797s) [noise=true , address=145.79.227.52:4445   , piece_id=4KJEI3BIXC7EUXNDWDYFGU2NLHPKOJUWI77MCVYZDE6QP3NPLF3A, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.590362772 (507h49m28.461445807s) [noise=true , address=145.79.227.47:4266   , piece_id=SVGPB5X3YLU2UIKUGJS4GUNMLKL6BXZPJRQ3G66T6JZT6MI6BSZQ, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.596782185 (507h49m28.46786523s) [noise=true , address=145.79.227.21:4475   , piece_id=5YKRLXVU7XSY3EC5PTQ4R3JLXAMOV2IIKAT65NYTLPJ4NUFVWTYA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.598067502 (507h49m28.469150537s) [noise=true , address=145.79.227.8:4330    , piece_id=GPZGJAV7FGQTP23ITFSSTIZWPC4ZMES6T74QZX452EIEYV4WTG7Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.600425994 (507h49m28.471509039s) [noise=true , address=145.79.227.50:4375   , piece_id=EZPMQA525ESJKSW6M365LF7GPW67RGM3NYWCJE3JQNVCQIRH433Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.604239915 (507h49m28.47532296s) [noise=true , address=145.79.227.45:4230   , piece_id=7RUBFVPQDHIUQ5HUXGGH2BA3BPQWHGN2SMAANXCP3UO5EVYGVIEQ, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.648156541 (507h49m28.519239576s) [noise=true , address=145.79.227.15:4875   , piece_id=EXHATTOQREEYUXA4ODVDSWQFDSYJUSPTTBT35DQYIARNSLFKDNFA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.850404066 (507h49m28.721487111s) [noise=true , address=77.90.31.14:4663     , piece_id=ENBFWB654DDFQFAS7CEYJUWBBCZW64OGGUZDRDCEP5MRNNDE7X3A, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.915554142 (507h49m28.786637177s) [noise=true , address=45.91.116.14:4189    , piece_id=EVWJBUTXLTGWSTY3KMZOIEIYIL6H4KFDYLON26DKINZIPSQ7UA3Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.941022299 (507h49m28.812105334s) [noise=true , address=45.91.116.13:4156    , piece_id=PMTLLDWJ3SSDLD6JUW55M6MQ667LCXWJMWQYRDBQGA5ZIBZOTL5Q, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.955655843 (507h49m28.826738878s) [noise=true , address=45.91.116.5:4300     , piece_id=MWVUKWESPIC2T2UM5HAP3OQJQ6NOIOIZK6VCZ5TJHDADO63BCVCA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: <nil>
Dec 26 19:18:28.955679017 (507h49m28.826762063s) [seg_pos={0 0}, upload=445627]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:18:28.955910448 (507h49m28.826993504s) [noise=true , address=145.79.227.4:4185    , piece_id=Y7SJVXPFSLK6RZZWRTKSIMTKEVSBT2N5ZI57GUCVMOVX4XMH7FIA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN): protocol: write tcp 188.166.225.170:42040->145.79.227.4:4185: use of closed network connection; write tcp 188.166.225.170:42040->145.79.227.4:4185: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42040->145.79.227.4:4185: use of closed network connection
Dec 26 19:18:28.955918734 (507h49m28.827001779s) [noise=true , address=145.79.227.4:4185    , piece_id=Y7SJVXPFSLK6RZZWRTKSIMTKEVSBT2N5ZI57GUCVMOVX4XMH7FIA, seg_pos={0 0}, upload=445627]: Long tail cancellation due to optimal threshold
Dec 26 19:18:28.956092866 (507h49m28.827175901s) [noise=true , address=145.79.227.30:4761   , piece_id=UWOPYRA6WTI75GRCAIKSYDPGZ7YEQPZAQP6ZM7ZIXPXQTDDE6YMA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 26 19:18:28.956130106 (507h49m28.827213151s) [noise=true , address=145.79.227.30:4761   , piece_id=UWOPYRA6WTI75GRCAIKSYDPGZ7YEQPZAQP6ZM7ZIXPXQTDDE6YMA, seg_pos={0 0}, upload=445627]: Long tail cancellation due to optimal threshold
Dec 26 19:18:28.956297095 (507h49m28.82738015s) [noise=true , address=77.90.31.12:4590     , piece_id=VMELCTDYY7FCQQXJH2NPPRAZ64YEIY2B3PIVOES4AHPPSQ2TUAWA, seg_pos={0 0}, upload=445627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:18:28.956307264 (507h49m28.827390319s) [noise=true , address=77.90.31.12:4590     , piece_id=VMELCTDYY7FCQQXJH2NPPRAZ64YEIY2B3PIVOES4AHPPSQ2TUAWA, seg_pos={0 0}, upload=445627]: Long tail cancellation due to optimal threshold
Dec 26 19:18:28.956330138 (507h49m28.827413183s) [seg_pos={0 0}, upload=445627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:18:28.956424948 (507h49m28.827507993s) [upload=445627]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:18:28.956431601 (507h49m28.827514666s) [upload=445627]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:18:29.259717791 (507h49m29.130800836s) [upload=445627]: Done uploading segments...
Dec 26 19:18:29.259737017 (507h49m29.130820052s) [upload=445627]: Upload finished. err: <nil>
Duration: 30.002396988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...X]
duration: 
30.003087512s
trace: 
https://jaeger.dev.storj.io/trace/5d7569c1fff810c5
FAIL (2025-12-26 19:17:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 19:17:22.580110613 (507h48m22.451193648s) [upload=445613]: Starting upload
Dec 26 19:17:22.580186407 (507h48m22.451269472s) [upload=445613]: Done starting upload
Dec 26 19:17:22.580272612 (507h48m22.451355647s) [upload=445613]: Uploading segments...
Dec 26 19:17:22.580470548 (507h48m22.451553593s) [upload=445613]: Got next segment. Inline: false
Dec 26 19:17:22.580511406 (507h48m22.451594451s) [upload=445613]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:17:22.580517598 (507h48m22.451600644s) [upload=445613]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:17:22.831904463 (507h48m22.702987508s) [seg_pos={0 0}, upload=445613]: Begin upload segment...
Dec 26 19:17:22.831965889 (507h48m22.703048924s) [seg_pos={0 0}, upload=445613]: Done begin upload segment.
Dec 26 19:17:22.832049128 (507h48m22.703132193s) [noise=true , address=145.79.227.17:4972   , piece_id=Y7546SOCADPAHMIHCJWX2QXZHNBQS5KXPDGEM6MHF5SRW3O4EU2Q, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.832068595 (507h48m22.70315164s) [noise=true , address=45.91.116.6:4973     , piece_id=T2QIAYEWQSDMRBVYP6NBJVHFV5X6WHISDULVSVLG6Q5IFEN75CLA, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.832150822 (507h48m22.703233867s) [noise=true , address=77.90.31.16:4736     , piece_id=5HZHYZEDX4QGUH5TTFEJD7WQ5YZFCCCMIWIVUADOTUTPMOA5CSOQ, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.832603193 (507h48m22.703686228s) [noise=true , address=145.79.227.29:4943   , piece_id=OXTFVJQ66JRUDU4TBZRACAS7K3Q57KKDLNVB3RQXETLRMY3J6XMA, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.832979991 (507h48m22.704063026s) [noise=true , address=145.79.227.50:4373   , piece_id=HTZVFRXDIOK2THILPRWPF227OSF5HQLKP3HAVM4BBNIPKX3VZTOQ, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.833328565 (507h48m22.70441161s) [noise=true , address=145.79.227.36:4519   , piece_id=4OTC4ECO3CSN6UPBCDMBHQ6GYUIKRQLIB53RRYUEQUOKDHZYAODA, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.833902036 (507h48m22.704985081s) [noise=true , address=145.79.227.21:4475   , piece_id=IM2NITU6YUO5JISV7DHC5FWL4N7PFCAHZVGTJPLPG2LHIWLN274A, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.834288552 (507h48m22.705371597s) [noise=true , address=45.91.116.5:4300     , piece_id=XBHVJEAOYLMHHES6HYJW7FI2THGPDQQUVRU3INJAXE75FIVXPAEQ, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.835366365 (507h48m22.7064494s) [noise=true , address=45.91.116.7:4010     , piece_id=E2UELKH2MJHIZA4MAUW6CVYEPF5ACTONI3Y4EAPYTOE34QHUDRRQ, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.836070876 (507h48m22.707153921s) [noise=true , address=145.79.227.37:4566   , piece_id=DECHKQLTGNJP6PPTMGVJHVINZU2ZUP3N5NESZBUKMWU4E3N2RRLQ, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.837665983 (507h48m22.708749018s) [noise=true , address=45.91.116.13:4156    , piece_id=4OTRZKFLXZ5QOVE6C5TYAS4X5NP46LAIE2YHBPB52TLQHMC6GNXQ, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.838905163 (507h48m22.709988208s) [noise=true , address=145.79.227.34:4906   , piece_id=K6XDZVFP5WOVIJBHJ2I66MSR63LVIPBBMFJIM4PDM66P7PQS3GEA, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.839975181 (507h48m22.711058216s) [noise=true , address=145.79.227.42:4123   , piece_id=WSG7OFS5ZG4FEXF32IN7TMGTPEE5NYMD7FMI2H25QRF22O7ZIV3Q, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.843199160 (507h48m22.714282205s) [noise=true , address=145.79.227.52:4446   , piece_id=XN5S723EXYOHRCKYMD3RVCJWTCNOZRKPJUXPTTEMSWRIHZ6UWO6A, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.845970467 (507h48m22.717053512s) [noise=true , address=45.91.116.14:4190    , piece_id=KPD22ZZYZYKKN3IFDBMSUZDEUJWFK6PBPLNK7LDBQJNTAGIAWK3Q, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:22.856966315 (507h48m22.72804938s) [noise=true , address=77.90.31.21:4880     , piece_id=EJLIXDTVJVRWNTQSWVF44BUVBKRHJY5GCBMCPNYBLNO7FWJEIV7Q, seg_pos={0 0}, upload=445613]: Uploading piece...
Dec 26 19:17:24.451893797 (507h48m24.322976842s) [upload=445613]: Next returned nil segment
Dec 26 19:17:24.451906852 (507h48m24.322989887s) [upload=445613]: Waiting for error group managing segments...
Dec 26 19:17:24.840151067 (507h48m24.711234112s) [noise=true , address=145.79.227.29:4943   , piece_id=OXTFVJQ66JRUDU4TBZRACAS7K3Q57KKDLNVB3RQXETLRMY3J6XMA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:24.850410993 (507h48m24.721494038s) [noise=true , address=145.79.227.21:4475   , piece_id=IM2NITU6YUO5JISV7DHC5FWL4N7PFCAHZVGTJPLPG2LHIWLN274A, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:24.857887349 (507h48m24.728970384s) [noise=true , address=145.79.227.36:4519   , piece_id=4OTC4ECO3CSN6UPBCDMBHQ6GYUIKRQLIB53RRYUEQUOKDHZYAODA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:24.861534053 (507h48m24.732617088s) [noise=true , address=145.79.227.42:4123   , piece_id=WSG7OFS5ZG4FEXF32IN7TMGTPEE5NYMD7FMI2H25QRF22O7ZIV3Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:24.863606520 (507h48m24.734689555s) [noise=true , address=145.79.227.50:4373   , piece_id=HTZVFRXDIOK2THILPRWPF227OSF5HQLKP3HAVM4BBNIPKX3VZTOQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:24.866588588 (507h48m24.737671633s) [noise=true , address=145.79.227.52:4446   , piece_id=XN5S723EXYOHRCKYMD3RVCJWTCNOZRKPJUXPTTEMSWRIHZ6UWO6A, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:24.873162546 (507h48m24.744245601s) [noise=true , address=145.79.227.37:4566   , piece_id=DECHKQLTGNJP6PPTMGVJHVINZU2ZUP3N5NESZBUKMWU4E3N2RRLQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.115439072 (507h48m24.986522108s) [noise=true , address=77.90.31.16:4736     , piece_id=5HZHYZEDX4QGUH5TTFEJD7WQ5YZFCCCMIWIVUADOTUTPMOA5CSOQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.116803822 (507h48m24.987886867s) [noise=true , address=45.91.116.13:4156    , piece_id=4OTRZKFLXZ5QOVE6C5TYAS4X5NP46LAIE2YHBPB52TLQHMC6GNXQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.136568887 (507h48m25.007651922s) [noise=true , address=45.91.116.7:4010     , piece_id=E2UELKH2MJHIZA4MAUW6CVYEPF5ACTONI3Y4EAPYTOE34QHUDRRQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.178459335 (507h48m25.04954237s) [noise=true , address=77.90.31.21:4880     , piece_id=EJLIXDTVJVRWNTQSWVF44BUVBKRHJY5GCBMCPNYBLNO7FWJEIV7Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.199009766 (507h48m25.070092801s) [noise=true , address=45.91.116.5:4300     , piece_id=XBHVJEAOYLMHHES6HYJW7FI2THGPDQQUVRU3INJAXE75FIVXPAEQ, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.219002777 (507h48m25.090085852s) [noise=true , address=45.91.116.14:4190    , piece_id=KPD22ZZYZYKKN3IFDBMSUZDEUJWFK6PBPLNK7LDBQJNTAGIAWK3Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: <nil>
Dec 26 19:17:25.219028695 (507h48m25.09011173s) [seg_pos={0 0}, upload=445613]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:17:25.219257652 (507h48m25.090340737s) [noise=true , address=145.79.227.17:4972   , piece_id=Y7546SOCADPAHMIHCJWX2QXZHNBQS5KXPDGEM6MHF5SRW3O4EU2Q, seg_pos={0 0}, upload=445613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: expected piece hash; context canceled; EOF
Dec 26 19:17:25.219268973 (507h48m25.090352028s) [noise=true , address=145.79.227.17:4972   , piece_id=Y7546SOCADPAHMIHCJWX2QXZHNBQS5KXPDGEM6MHF5SRW3O4EU2Q, seg_pos={0 0}, upload=445613]: Long tail cancellation due to optimal threshold
Dec 26 19:17:25.219347252 (507h48m25.090430287s) [noise=true , address=145.79.227.34:4906   , piece_id=K6XDZVFP5WOVIJBHJ2I66MSR63LVIPBBMFJIM4PDM66P7PQS3GEA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:17:25.219357322 (507h48m25.090440357s) [noise=true , address=145.79.227.34:4906   , piece_id=K6XDZVFP5WOVIJBHJ2I66MSR63LVIPBBMFJIM4PDM66P7PQS3GEA, seg_pos={0 0}, upload=445613]: Long tail cancellation due to optimal threshold
Dec 26 19:17:25.219587790 (507h48m25.090670845s) [noise=true , address=45.91.116.6:4973     , piece_id=T2QIAYEWQSDMRBVYP6NBJVHFV5X6WHISDULVSVLG6Q5IFEN75CLA, seg_pos={0 0}, upload=445613]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 26 19:17:25.219611165 (507h48m25.09069421s) [noise=true , address=45.91.116.6:4973     , piece_id=T2QIAYEWQSDMRBVYP6NBJVHFV5X6WHISDULVSVLG6Q5IFEN75CLA, seg_pos={0 0}, upload=445613]: Long tail cancellation due to optimal threshold
Dec 26 19:17:25.219634419 (507h48m25.090717454s) [seg_pos={0 0}, upload=445613]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:17:25.219720092 (507h48m25.090803127s) [upload=445613]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:17:25.219727967 (507h48m25.090811002s) [upload=445613]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:17:25.516821405 (507h48m25.38790445s) [upload=445613]: Done uploading segments...
Dec 26 19:17:25.516848556 (507h48m25.387931631s) [upload=445613]: Upload finished. err: <nil>
Duration: 30.003100208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X...]
duration: 
26.053880201s
trace: 
https://jaeger.dev.storj.io/trace/8cbaebf4bc4ac12
FAIL (2025-12-26 19:17:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q) context deadline exceeded
======LOGS======
 (507h45m39.539934276s) [upload=445569]: Done starting upload
Dec 26 19:14:39.668953175 (507h45m39.54003622s) [upload=445569]: Uploading segments...
Dec 26 19:14:39.669098882 (507h45m39.540181917s) [upload=445569]: Got next segment. Inline: false
Dec 26 19:14:39.669131354 (507h45m39.540214389s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:14:39.669138347 (507h45m39.540221382s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:14:39.919054782 (507h45m39.790137817s) [seg_pos={0 0}, upload=445569]: Begin upload segment...
Dec 26 19:14:39.919108674 (507h45m39.790191709s) [seg_pos={0 0}, upload=445569]: Done begin upload segment.
Dec 26 19:14:39.919135595 (507h45m39.79021863s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919154241 (507h45m39.790237286s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919204817 (507h45m39.790287852s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919472367 (507h45m39.790555412s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919769612 (507h45m39.790852648s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920059435 (507h45m39.79114247s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920325922 (507h45m39.791408957s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920744308 (507h45m39.791827344s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921114644 (507h45m39.792197689s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921466324 (507h45m39.792549369s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921896022 (507h45m39.792979068s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923066100 (507h45m39.794149145s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923152836 (507h45m39.794235881s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.924242731 (507h45m39.795325766s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.925507038 (507h45m39.796590083s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.927865880 (507h45m39.798948915s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:41.639443473 (507h45m41.510526508s) [upload=445569]: Next returned nil segment
Dec 26 19:14:41.639464642 (507h45m41.510547677s) [upload=445569]: Waiting for error group managing segments...
Dec 26 19:14:41.926143951 (507h45m41.797226986s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.939547924 (507h45m41.810630959s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940037276 (507h45m41.811120321s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940433921 (507h45m41.811516956s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.941067779 (507h45m41.812150814s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.942830555 (507h45m41.81391359s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.957594908 (507h45m41.828677944s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958160777 (507h45m41.829243812s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958603559 (507h45m41.829686604s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.963562983 (507h45m41.834646018s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.038967997 (507h45m41.910051042s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.167135100 (507h45m42.038218135s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205466509 (507h45m42.076549544s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205483141 (507h45m42.076566177s) [seg_pos={0 0}, upload=445569]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:14:42.205645099 (507h45m42.076728134s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:14:42.205650480 (507h45m42.076733515s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.205763836 (507h45m42.076846871s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection
Dec 26 19:14:42.205769326 (507h45m42.076852361s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206093253 (507h45m42.077176298s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:51406->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 26 19:14:42.206110065 (507h45m42.0771931s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206152786 (507h45m42.077235821s) [seg_pos={0 0}, upload=445569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:14:42.206317099 (507h45m42.077400135s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:14:42.206322900 (507h45m42.077405935s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:14:42.505831747 (507h45m42.376914782s) [upload=445569]: Done uploading segments...
Dec 26 19:14:42.505853869 (507h45m42.376936905s) [upload=445569]: Upload finished. err: <nil>
Duration: 26.053905788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X..]
duration: 
5.895963294s
trace: 
https://jaeger.dev.storj.io/trace/8cbaebf4bc4ac12
FAIL (2025-12-26 19:15:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q) context deadline exceeded
======LOGS======
 (507h45m39.539934276s) [upload=445569]: Done starting upload
Dec 26 19:14:39.668953175 (507h45m39.54003622s) [upload=445569]: Uploading segments...
Dec 26 19:14:39.669098882 (507h45m39.540181917s) [upload=445569]: Got next segment. Inline: false
Dec 26 19:14:39.669131354 (507h45m39.540214389s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:14:39.669138347 (507h45m39.540221382s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:14:39.919054782 (507h45m39.790137817s) [seg_pos={0 0}, upload=445569]: Begin upload segment...
Dec 26 19:14:39.919108674 (507h45m39.790191709s) [seg_pos={0 0}, upload=445569]: Done begin upload segment.
Dec 26 19:14:39.919135595 (507h45m39.79021863s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919154241 (507h45m39.790237286s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919204817 (507h45m39.790287852s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919472367 (507h45m39.790555412s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919769612 (507h45m39.790852648s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920059435 (507h45m39.79114247s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920325922 (507h45m39.791408957s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920744308 (507h45m39.791827344s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921114644 (507h45m39.792197689s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921466324 (507h45m39.792549369s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921896022 (507h45m39.792979068s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923066100 (507h45m39.794149145s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923152836 (507h45m39.794235881s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.924242731 (507h45m39.795325766s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.925507038 (507h45m39.796590083s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.927865880 (507h45m39.798948915s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:41.639443473 (507h45m41.510526508s) [upload=445569]: Next returned nil segment
Dec 26 19:14:41.639464642 (507h45m41.510547677s) [upload=445569]: Waiting for error group managing segments...
Dec 26 19:14:41.926143951 (507h45m41.797226986s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.939547924 (507h45m41.810630959s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940037276 (507h45m41.811120321s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940433921 (507h45m41.811516956s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.941067779 (507h45m41.812150814s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.942830555 (507h45m41.81391359s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.957594908 (507h45m41.828677944s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958160777 (507h45m41.829243812s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958603559 (507h45m41.829686604s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.963562983 (507h45m41.834646018s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.038967997 (507h45m41.910051042s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.167135100 (507h45m42.038218135s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205466509 (507h45m42.076549544s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205483141 (507h45m42.076566177s) [seg_pos={0 0}, upload=445569]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:14:42.205645099 (507h45m42.076728134s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:14:42.205650480 (507h45m42.076733515s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.205763836 (507h45m42.076846871s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection
Dec 26 19:14:42.205769326 (507h45m42.076852361s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206093253 (507h45m42.077176298s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:51406->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 26 19:14:42.206110065 (507h45m42.0771931s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206152786 (507h45m42.077235821s) [seg_pos={0 0}, upload=445569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:14:42.206317099 (507h45m42.077400135s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:14:42.206322900 (507h45m42.077405935s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:14:42.505831747 (507h45m42.376914782s) [upload=445569]: Done uploading segments...
Dec 26 19:14:42.505853869 (507h45m42.376936905s) [upload=445569]: Upload finished. err: <nil>
Duration: 5.895990356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X.]
duration: 
6.165463319s
trace: 
https://jaeger.dev.storj.io/trace/8cbaebf4bc4ac12
FAIL (2025-12-26 19:15:15 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q) context deadline exceeded
======LOGS======
 (507h45m39.539934276s) [upload=445569]: Done starting upload
Dec 26 19:14:39.668953175 (507h45m39.54003622s) [upload=445569]: Uploading segments...
Dec 26 19:14:39.669098882 (507h45m39.540181917s) [upload=445569]: Got next segment. Inline: false
Dec 26 19:14:39.669131354 (507h45m39.540214389s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:14:39.669138347 (507h45m39.540221382s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:14:39.919054782 (507h45m39.790137817s) [seg_pos={0 0}, upload=445569]: Begin upload segment...
Dec 26 19:14:39.919108674 (507h45m39.790191709s) [seg_pos={0 0}, upload=445569]: Done begin upload segment.
Dec 26 19:14:39.919135595 (507h45m39.79021863s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919154241 (507h45m39.790237286s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919204817 (507h45m39.790287852s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919472367 (507h45m39.790555412s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919769612 (507h45m39.790852648s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920059435 (507h45m39.79114247s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920325922 (507h45m39.791408957s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920744308 (507h45m39.791827344s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921114644 (507h45m39.792197689s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921466324 (507h45m39.792549369s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921896022 (507h45m39.792979068s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923066100 (507h45m39.794149145s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923152836 (507h45m39.794235881s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.924242731 (507h45m39.795325766s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.925507038 (507h45m39.796590083s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.927865880 (507h45m39.798948915s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:41.639443473 (507h45m41.510526508s) [upload=445569]: Next returned nil segment
Dec 26 19:14:41.639464642 (507h45m41.510547677s) [upload=445569]: Waiting for error group managing segments...
Dec 26 19:14:41.926143951 (507h45m41.797226986s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.939547924 (507h45m41.810630959s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940037276 (507h45m41.811120321s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940433921 (507h45m41.811516956s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.941067779 (507h45m41.812150814s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.942830555 (507h45m41.81391359s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.957594908 (507h45m41.828677944s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958160777 (507h45m41.829243812s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958603559 (507h45m41.829686604s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.963562983 (507h45m41.834646018s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.038967997 (507h45m41.910051042s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.167135100 (507h45m42.038218135s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205466509 (507h45m42.076549544s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205483141 (507h45m42.076566177s) [seg_pos={0 0}, upload=445569]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:14:42.205645099 (507h45m42.076728134s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:14:42.205650480 (507h45m42.076733515s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.205763836 (507h45m42.076846871s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection
Dec 26 19:14:42.205769326 (507h45m42.076852361s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206093253 (507h45m42.077176298s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:51406->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 26 19:14:42.206110065 (507h45m42.0771931s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206152786 (507h45m42.077235821s) [seg_pos={0 0}, upload=445569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:14:42.206317099 (507h45m42.077400135s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:14:42.206322900 (507h45m42.077405935s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:14:42.505831747 (507h45m42.376914782s) [upload=445569]: Done uploading segments...
Dec 26 19:14:42.505853869 (507h45m42.376936905s) [upload=445569]: Upload finished. err: <nil>
Duration: 6.165483868s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.X]
duration: 
30.003194136s
trace: 
https://jaeger.dev.storj.io/trace/8cbaebf4bc4ac12
FAIL (2025-12-26 19:15:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q) context deadline exceeded
======LOGS======
 (507h45m39.539934276s) [upload=445569]: Done starting upload
Dec 26 19:14:39.668953175 (507h45m39.54003622s) [upload=445569]: Uploading segments...
Dec 26 19:14:39.669098882 (507h45m39.540181917s) [upload=445569]: Got next segment. Inline: false
Dec 26 19:14:39.669131354 (507h45m39.540214389s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:14:39.669138347 (507h45m39.540221382s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:14:39.919054782 (507h45m39.790137817s) [seg_pos={0 0}, upload=445569]: Begin upload segment...
Dec 26 19:14:39.919108674 (507h45m39.790191709s) [seg_pos={0 0}, upload=445569]: Done begin upload segment.
Dec 26 19:14:39.919135595 (507h45m39.79021863s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919154241 (507h45m39.790237286s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919204817 (507h45m39.790287852s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919472367 (507h45m39.790555412s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.919769612 (507h45m39.790852648s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920059435 (507h45m39.79114247s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920325922 (507h45m39.791408957s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.920744308 (507h45m39.791827344s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921114644 (507h45m39.792197689s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921466324 (507h45m39.792549369s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.921896022 (507h45m39.792979068s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923066100 (507h45m39.794149145s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.923152836 (507h45m39.794235881s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.924242731 (507h45m39.795325766s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.925507038 (507h45m39.796590083s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:39.927865880 (507h45m39.798948915s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Uploading piece...
Dec 26 19:14:41.639443473 (507h45m41.510526508s) [upload=445569]: Next returned nil segment
Dec 26 19:14:41.639464642 (507h45m41.510547677s) [upload=445569]: Waiting for error group managing segments...
Dec 26 19:14:41.926143951 (507h45m41.797226986s) [noise=true , address=145.79.227.38:4977   , piece_id=BSEQX4A2U5NS6WIHLGYASS5AGYTVIQ2OTBEIYY4FPT6RTQHFSYGA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.939547924 (507h45m41.810630959s) [noise=true , address=145.79.227.36:4518   , piece_id=WO4BHISF4VXIKQ3ZFLMJ3DFGDR7DUE3X6B7F252FBDL5P7S5HPHQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940037276 (507h45m41.811120321s) [noise=true , address=145.79.227.8:4330    , piece_id=GZJZ3GECAV23L5VE3XMIHD2Y3ES3BZTZG7VXLQ2YWEJOHORHXSJQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.940433921 (507h45m41.811516956s) [noise=true , address=145.79.227.16:4910   , piece_id=6IRCORKQ2CVFFVWA2GK7I7TVQARHNHYUGZA47GY6K3BCCDBGYQVA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.941067779 (507h45m41.812150814s) [noise=true , address=145.79.227.45:4229   , piece_id=GJP6EB672G2D5D2A3LHSH6RXK4UJUEIVH4JLVBIJ5YCELQHVNSZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.942830555 (507h45m41.81391359s) [noise=true , address=145.79.227.22:4509   , piece_id=LEKFN54HGK35DTRTUYAKXKF5JJQ23DAZLSO2HALBZ6MYBEN5AZFQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.957594908 (507h45m41.828677944s) [noise=true , address=145.79.227.50:4374   , piece_id=7O6Q3A65K2KWXTNGH7BHVU6YC2UO7PWK73DGEF4ZUGA3WI7XXEUQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958160777 (507h45m41.829243812s) [noise=true , address=145.79.227.3:4149    , piece_id=MQBHY7DFFTMPDBJWDOCUUPSZSY46XDDTTV2O5CBUQ54R5DLCTEZA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.958603559 (507h45m41.829686604s) [noise=true , address=145.79.227.39:4013   , piece_id=ILITPP6WC2OHVZ5P2L6WCT5CWQF5DED6UAATACRDRBO5M4AOIL7Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:41.963562983 (507h45m41.834646018s) [noise=true , address=145.79.227.51:4411   , piece_id=BPVRGE4O2HXTY7VCGVNILDJ5QOSUO44DLENXM4OKP75WXE7GANTA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.038967997 (507h45m41.910051042s) [noise=true , address=145.79.227.14:4839   , piece_id=I3SXTKRU2AIDPGT5XAHDM7H44H57IR4MYMAOEJQWZ4IJG6Y4ZRXQ, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.167135100 (507h45m42.038218135s) [noise=true , address=45.91.116.14:4191    , piece_id=MEZUGSOURBEALY34U5BBBSKMAXOEIZCCNB6VN5MSDZ24QRKZLJ2Q, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205466509 (507h45m42.076549544s) [noise=true , address=77.90.31.21:4878     , piece_id=SQX7UN7TWFVGMXNTYV7P34EDQXIWE2MOZ4URA4FVT5KHVSS64PJA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: <nil>
Dec 26 19:14:42.205483141 (507h45m42.076566177s) [seg_pos={0 0}, upload=445569]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:14:42.205645099 (507h45m42.076728134s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF
Dec 26 19:14:42.205650480 (507h45m42.076733515s) [noise=true , address=77.90.31.12:4590     , piece_id=NVVFMA76YPEI4E5SJDXUDJNTMS2JRJ74ANYQHVPRGVDBLKHBT3UA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.205763836 (507h45m42.076846871s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41748->45.91.116.4:4937: use of closed network connection
Dec 26 19:14:42.205769326 (507h45m42.076852361s) [noise=true , address=45.91.116.4:4937     , piece_id=GRARRBQCAVXC62M32ND7R4FJTCRGZ6BGPBA7TQUJ6BVHKS3HTGNA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206093253 (507h45m42.077176298s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: read tcp 188.166.225.170:51406->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 26 19:14:42.206110065 (507h45m42.0771931s) [noise=true , address=45.91.116.10:4082    , piece_id=4KZT64WH7HWZGJEJ23PSWKUNF73RZHUGPUNNQ6TD2V4RPLG5VDXA, seg_pos={0 0}, upload=445569]: Long tail cancellation due to optimal threshold
Dec 26 19:14:42.206152786 (507h45m42.077235821s) [seg_pos={0 0}, upload=445569]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:14:42.206317099 (507h45m42.077400135s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:14:42.206322900 (507h45m42.077405935s) [upload=445569]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:14:42.505831747 (507h45m42.376914782s) [upload=445569]: Done uploading segments...
Dec 26 19:14:42.505853869 (507h45m42.376936905s) [upload=445569]: Upload finished. err: <nil>
Duration: 30.003220838s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX.]
duration: 
6.24177117s
trace: 
https://jaeger.dev.storj.io/trace/1a9364135555cc37
FAIL (2025-12-26 19:13:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: WGNCIYTKOMXU6IBC2EEVCXHVI3OAOJTRWQC5QI4463PHAHHZWYSQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: DDLDBN5BBEDAQOSQ6ICOKUBYDESAJYJTRQPPRN3PHACMHFHITQ4A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2IAO42SZZH7PUAY5IAKN23WV2G7BCSVRKKBX56K3U3RJRMLZM4NQ) context deadline exceeded
======LOGS======
Dec 26 19:13:12.972469057 (507h44m12.843552092s) [upload=445548]: Starting upload
Dec 26 19:13:12.972556585 (507h44m12.84363963s) [upload=445548]: Done starting upload
Dec 26 19:13:12.972722320 (507h44m12.843805355s) [upload=445548]: Uploading segments...
Dec 26 19:13:12.972811850 (507h44m12.843894895s) [upload=445548]: Got next segment. Inline: false
Dec 26 19:13:12.972852758 (507h44m12.843935793s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:13:12.972860063 (507h44m12.843943108s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:13:13.222796345 (507h44m13.09387941s) [seg_pos={0 0}, upload=445548]: Begin upload segment...
Dec 26 19:13:13.222867951 (507h44m13.093951006s) [seg_pos={0 0}, upload=445548]: Done begin upload segment.
Dec 26 19:13:13.222888581 (507h44m13.093971616s) [noise=true , address=145.79.227.37:4567   , piece_id=4KBU3TXGRLI7CFTP52JWK475Q5OX4DZDVJZ65UWJTJ3Y5XBC4L7Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.222916763 (507h44m13.093999809s) [noise=true , address=145.79.227.50:4374   , piece_id=IS46YQVI6SX7KVOXKFDQ6YK2BIVN3QBJOPNISSS7Q4JSWBRAV33Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.223481129 (507h44m13.094564174s) [noise=true , address=77.90.31.13:4629     , piece_id=2IAO42SZZH7PUAY5IAKN23WV2G7BCSVRKKBX56K3U3RJRMLZM4NQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.224018953 (507h44m13.095102088s) [noise=true , address=145.79.227.12:4767   , piece_id=TAGY6HTNQGG3EC5YU3QDUZHDS3S32OHBOTFLHF26F5DY7BWV3D2Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.224628063 (507h44m13.095711108s) [noise=true , address=45.91.116.9:4262     , piece_id=KNOXYH3SHMBH5IUXWD3SZ676AL67MYLOC6ELK62TY2Y455NGW7OA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.225130930 (507h44m13.096214005s) [noise=true , address=77.90.31.17:4773     , piece_id=64OJWKGO33YZWY3HJWO7KGC2DRZ346BDQ5NC2D2UNBQJZQHDZKTQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.225620492 (507h44m13.096703537s) [noise=true , address=145.79.227.31:4799   , piece_id=KRU5HN7QK7TF3ZP5YKEYYNIKGE57SWCM72753R3BDIIP5QSORUSA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.227439486 (507h44m13.098522531s) [noise=true , address=145.79.227.51:4409   , piece_id=HVU2VMRB2R6YD53WYYONKBHWFM6AT4DRBPVM3JDJ3L32DTAI4HCQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.229327501 (507h44m13.100410557s) [noise=true , address=77.90.31.21:4878     , piece_id=WGNCIYTKOMXU6IBC2EEVCXHVI3OAOJTRWQC5QI4463PHAHHZWYSQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.231116578 (507h44m13.102199613s) [noise=true , address=77.90.31.16:4735     , piece_id=DDLDBN5BBEDAQOSQ6ICOKUBYDESAJYJTRQPPRN3PHACMHFHITQ4A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.232481036 (507h44m13.103564081s) [noise=true , address=145.79.227.45:4230   , piece_id=NMNL6R5GZUYCF2P5YNV5ZEFSAB2QKYNCK62MQ2OB5F7Q5RPQ43IA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.232944880 (507h44m13.104027915s) [noise=true , address=45.91.116.10:4083    , piece_id=PD3LCV3LGK6T6XWG73YK4H3UR3OAPGV652N5TFMQJJW7HPL6ENTA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.234168590 (507h44m13.105251625s) [noise=true , address=145.79.227.46:4602   , piece_id=2KTN6XFIWMZ5VYRHZH44UFBHGK2B7GSZMK5MCLFGW5LWVOP2SN2Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.235942207 (507h44m13.107025252s) [noise=true , address=45.91.116.5:4300     , piece_id=IZFHD6SPFWWNRPODPAF4Y755GW5YUSNSAFUV7CUOQXN36S5QBO3A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.236519806 (507h44m13.107602851s) [noise=true , address=145.79.227.36:4519   , piece_id=UPEDCJMPUFVJHQYP5HZUY6WCYNGQ3HNDFMOXD6N3H7EIVR6THB2A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.240717490 (507h44m13.111800535s) [noise=true , address=45.91.116.4:4940     , piece_id=X2SG52W4CKEXSBZ7GBFV3FSTZFEROWH64BVFMZVUFBQDOWHPWR4A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:14.943136254 (507h44m14.814219289s) [upload=445548]: Next returned nil segment
Dec 26 19:13:14.943225134 (507h44m14.814308169s) [upload=445548]: Waiting for error group managing segments...
Dec 26 19:13:15.241469233 (507h44m15.112552268s) [noise=true , address=145.79.227.31:4799   , piece_id=KRU5HN7QK7TF3ZP5YKEYYNIKGE57SWCM72753R3BDIIP5QSORUSA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.244780719 (507h44m15.115863764s) [noise=true , address=145.79.227.45:4230   , piece_id=NMNL6R5GZUYCF2P5YNV5ZEFSAB2QKYNCK62MQ2OB5F7Q5RPQ43IA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.247384377 (507h44m15.118467412s) [noise=true , address=145.79.227.51:4409   , piece_id=HVU2VMRB2R6YD53WYYONKBHWFM6AT4DRBPVM3JDJ3L32DTAI4HCQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.247795860 (507h44m15.118878905s) [noise=true , address=145.79.227.37:4567   , piece_id=4KBU3TXGRLI7CFTP52JWK475Q5OX4DZDVJZ65UWJTJ3Y5XBC4L7Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.249902992 (507h44m15.120986027s) [noise=true , address=145.79.227.46:4602   , piece_id=2KTN6XFIWMZ5VYRHZH44UFBHGK2B7GSZMK5MCLFGW5LWVOP2SN2Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.252625236 (507h44m15.123708271s) [noise=true , address=145.79.227.50:4374   , piece_id=IS46YQVI6SX7KVOXKFDQ6YK2BIVN3QBJOPNISSS7Q4JSWBRAV33Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.256490065 (507h44m15.1275731s) [noise=true , address=145.79.227.36:4519   , piece_id=UPEDCJMPUFVJHQYP5HZUY6WCYNGQ3HNDFMOXD6N3H7EIVR6THB2A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.267997037 (507h44m15.139080072s) [noise=true , address=145.79.227.12:4767   , piece_id=TAGY6HTNQGG3EC5YU3QDUZHDS3S32OHBOTFLHF26F5DY7BWV3D2Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.426652354 (507h44m15.297735399s) [noise=true , address=77.90.31.16:4735     , piece_id=DDLDBN5BBEDAQOSQ6ICOKUBYDESAJYJTRQPPRN3PHACMHFHITQ4A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.448050911 (507h44m15.319133946s) [noise=true , address=77.90.31.13:4629     , piece_id=2IAO42SZZH7PUAY5IAKN23WV2G7BCSVRKKBX56K3U3RJRMLZM4NQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.502573561 (507h44m15.373656606s) [noise=true , address=45.91.116.9:4262     , piece_id=KNOXYH3SHMBH5IUXWD3SZ676AL67MYLOC6ELK62TY2Y455NGW7OA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.520677514 (507h44m15.391760549s) [noise=true , address=45.91.116.4:4940     , piece_id=X2SG52W4CKEXSBZ7GBFV3FSTZFEROWH64BVFMZVUFBQDOWHPWR4A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.545196564 (507h44m15.416279609s) [noise=true , address=77.90.31.21:4878     , piece_id=WGNCIYTKOMXU6IBC2EEVCXHVI3OAOJTRWQC5QI4463PHAHHZWYSQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.545223244 (507h44m15.416306289s) [seg_pos={0 0}, upload=445548]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:13:15.545439286 (507h44m15.416522321s) [noise=true , address=45.91.116.5:4300     , piece_id=IZFHD6SPFWWNRPODPAF4Y755GW5YUSNSAFUV7CUOQXN36S5QBO3A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 26 19:13:15.545453713 (507h44m15.416536788s) [noise=true , address=45.91.116.5:4300     , piece_id=IZFHD6SPFWWNRPODPAF4Y755GW5YUSNSAFUV7CUOQXN36S5QBO3A, seg_pos={0 0}, upload=445548]: Long tail cancellation due to optimal threshold
Dec 26 19:13:15.545651981 (507h44m15.416735026s) [noise=true , address=77.90.31.17:4773     , piece_id=64OJWKGO33YZWY3HJWO7KGC2DRZ346BDQ5NC2D2UNBQJZQHDZKTQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:51854->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:51854->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51854->77.90.31.17:4773: use of closed network connection
Dec 26 19:13:15.545659505 (507h44m15.41674254s) [noise=true , address=77.90.31.17:4773     , piece_id=64OJWKGO33YZWY3HJWO7KGC2DRZ346BDQ5NC2D2UNBQJZQHDZKTQ, seg_pos={0 0}, upload=445548]: Long tail cancellation due to optimal threshold
Dec 26 19:13:15.545872490 (507h44m15.416955545s) [noise=true , address=45.91.116.10:4083    , piece_id=PD3LCV3LGK6T6XWG73YK4H3UR3OAPGV652N5TFMQJJW7HPL6ENTA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 26 19:13:15.545895023 (507h44m15.416978069s) [noise=true , address=45.91.116.10:4083    , piece_id=PD3LCV3LGK6T6XWG73YK4H3UR3OAPGV652N5TFMQJJW7HPL6ENTA, seg_pos={0 0}, upload=445548]: Long tail cancellation due to optimal threshold
Dec 26 19:13:15.545968383 (507h44m15.417051428s) [seg_pos={0 0}, upload=445548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:13:15.546087790 (507h44m15.417170835s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:13:15.546095055 (507h44m15.4171781s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:13:15.854498168 (507h44m15.725581213s) [upload=445548]: Done uploading segments...
Dec 26 19:13:15.854517415 (507h44m15.72560045s) [upload=445548]: Upload finished. err: <nil>
Duration: 6.241785217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XXX]
duration: 
30.00329437s
trace: 
https://jaeger.dev.storj.io/trace/1a9364135555cc37
FAIL (2025-12-26 19:13:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: WGNCIYTKOMXU6IBC2EEVCXHVI3OAOJTRWQC5QI4463PHAHHZWYSQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: DDLDBN5BBEDAQOSQ6ICOKUBYDESAJYJTRQPPRN3PHACMHFHITQ4A) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: 2IAO42SZZH7PUAY5IAKN23WV2G7BCSVRKKBX56K3U3RJRMLZM4NQ) context deadline exceeded
======LOGS======
Dec 26 19:13:12.972469057 (507h44m12.843552092s) [upload=445548]: Starting upload
Dec 26 19:13:12.972556585 (507h44m12.84363963s) [upload=445548]: Done starting upload
Dec 26 19:13:12.972722320 (507h44m12.843805355s) [upload=445548]: Uploading segments...
Dec 26 19:13:12.972811850 (507h44m12.843894895s) [upload=445548]: Got next segment. Inline: false
Dec 26 19:13:12.972852758 (507h44m12.843935793s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:13:12.972860063 (507h44m12.843943108s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:13:13.222796345 (507h44m13.09387941s) [seg_pos={0 0}, upload=445548]: Begin upload segment...
Dec 26 19:13:13.222867951 (507h44m13.093951006s) [seg_pos={0 0}, upload=445548]: Done begin upload segment.
Dec 26 19:13:13.222888581 (507h44m13.093971616s) [noise=true , address=145.79.227.37:4567   , piece_id=4KBU3TXGRLI7CFTP52JWK475Q5OX4DZDVJZ65UWJTJ3Y5XBC4L7Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.222916763 (507h44m13.093999809s) [noise=true , address=145.79.227.50:4374   , piece_id=IS46YQVI6SX7KVOXKFDQ6YK2BIVN3QBJOPNISSS7Q4JSWBRAV33Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.223481129 (507h44m13.094564174s) [noise=true , address=77.90.31.13:4629     , piece_id=2IAO42SZZH7PUAY5IAKN23WV2G7BCSVRKKBX56K3U3RJRMLZM4NQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.224018953 (507h44m13.095102088s) [noise=true , address=145.79.227.12:4767   , piece_id=TAGY6HTNQGG3EC5YU3QDUZHDS3S32OHBOTFLHF26F5DY7BWV3D2Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.224628063 (507h44m13.095711108s) [noise=true , address=45.91.116.9:4262     , piece_id=KNOXYH3SHMBH5IUXWD3SZ676AL67MYLOC6ELK62TY2Y455NGW7OA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.225130930 (507h44m13.096214005s) [noise=true , address=77.90.31.17:4773     , piece_id=64OJWKGO33YZWY3HJWO7KGC2DRZ346BDQ5NC2D2UNBQJZQHDZKTQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.225620492 (507h44m13.096703537s) [noise=true , address=145.79.227.31:4799   , piece_id=KRU5HN7QK7TF3ZP5YKEYYNIKGE57SWCM72753R3BDIIP5QSORUSA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.227439486 (507h44m13.098522531s) [noise=true , address=145.79.227.51:4409   , piece_id=HVU2VMRB2R6YD53WYYONKBHWFM6AT4DRBPVM3JDJ3L32DTAI4HCQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.229327501 (507h44m13.100410557s) [noise=true , address=77.90.31.21:4878     , piece_id=WGNCIYTKOMXU6IBC2EEVCXHVI3OAOJTRWQC5QI4463PHAHHZWYSQ, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.231116578 (507h44m13.102199613s) [noise=true , address=77.90.31.16:4735     , piece_id=DDLDBN5BBEDAQOSQ6ICOKUBYDESAJYJTRQPPRN3PHACMHFHITQ4A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.232481036 (507h44m13.103564081s) [noise=true , address=145.79.227.45:4230   , piece_id=NMNL6R5GZUYCF2P5YNV5ZEFSAB2QKYNCK62MQ2OB5F7Q5RPQ43IA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.232944880 (507h44m13.104027915s) [noise=true , address=45.91.116.10:4083    , piece_id=PD3LCV3LGK6T6XWG73YK4H3UR3OAPGV652N5TFMQJJW7HPL6ENTA, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.234168590 (507h44m13.105251625s) [noise=true , address=145.79.227.46:4602   , piece_id=2KTN6XFIWMZ5VYRHZH44UFBHGK2B7GSZMK5MCLFGW5LWVOP2SN2Q, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.235942207 (507h44m13.107025252s) [noise=true , address=45.91.116.5:4300     , piece_id=IZFHD6SPFWWNRPODPAF4Y755GW5YUSNSAFUV7CUOQXN36S5QBO3A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.236519806 (507h44m13.107602851s) [noise=true , address=145.79.227.36:4519   , piece_id=UPEDCJMPUFVJHQYP5HZUY6WCYNGQ3HNDFMOXD6N3H7EIVR6THB2A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:13.240717490 (507h44m13.111800535s) [noise=true , address=45.91.116.4:4940     , piece_id=X2SG52W4CKEXSBZ7GBFV3FSTZFEROWH64BVFMZVUFBQDOWHPWR4A, seg_pos={0 0}, upload=445548]: Uploading piece...
Dec 26 19:13:14.943136254 (507h44m14.814219289s) [upload=445548]: Next returned nil segment
Dec 26 19:13:14.943225134 (507h44m14.814308169s) [upload=445548]: Waiting for error group managing segments...
Dec 26 19:13:15.241469233 (507h44m15.112552268s) [noise=true , address=145.79.227.31:4799   , piece_id=KRU5HN7QK7TF3ZP5YKEYYNIKGE57SWCM72753R3BDIIP5QSORUSA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.244780719 (507h44m15.115863764s) [noise=true , address=145.79.227.45:4230   , piece_id=NMNL6R5GZUYCF2P5YNV5ZEFSAB2QKYNCK62MQ2OB5F7Q5RPQ43IA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.247384377 (507h44m15.118467412s) [noise=true , address=145.79.227.51:4409   , piece_id=HVU2VMRB2R6YD53WYYONKBHWFM6AT4DRBPVM3JDJ3L32DTAI4HCQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.247795860 (507h44m15.118878905s) [noise=true , address=145.79.227.37:4567   , piece_id=4KBU3TXGRLI7CFTP52JWK475Q5OX4DZDVJZ65UWJTJ3Y5XBC4L7Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.249902992 (507h44m15.120986027s) [noise=true , address=145.79.227.46:4602   , piece_id=2KTN6XFIWMZ5VYRHZH44UFBHGK2B7GSZMK5MCLFGW5LWVOP2SN2Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.252625236 (507h44m15.123708271s) [noise=true , address=145.79.227.50:4374   , piece_id=IS46YQVI6SX7KVOXKFDQ6YK2BIVN3QBJOPNISSS7Q4JSWBRAV33Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.256490065 (507h44m15.1275731s) [noise=true , address=145.79.227.36:4519   , piece_id=UPEDCJMPUFVJHQYP5HZUY6WCYNGQ3HNDFMOXD6N3H7EIVR6THB2A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.267997037 (507h44m15.139080072s) [noise=true , address=145.79.227.12:4767   , piece_id=TAGY6HTNQGG3EC5YU3QDUZHDS3S32OHBOTFLHF26F5DY7BWV3D2Q, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.426652354 (507h44m15.297735399s) [noise=true , address=77.90.31.16:4735     , piece_id=DDLDBN5BBEDAQOSQ6ICOKUBYDESAJYJTRQPPRN3PHACMHFHITQ4A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.448050911 (507h44m15.319133946s) [noise=true , address=77.90.31.13:4629     , piece_id=2IAO42SZZH7PUAY5IAKN23WV2G7BCSVRKKBX56K3U3RJRMLZM4NQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.502573561 (507h44m15.373656606s) [noise=true , address=45.91.116.9:4262     , piece_id=KNOXYH3SHMBH5IUXWD3SZ676AL67MYLOC6ELK62TY2Y455NGW7OA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.520677514 (507h44m15.391760549s) [noise=true , address=45.91.116.4:4940     , piece_id=X2SG52W4CKEXSBZ7GBFV3FSTZFEROWH64BVFMZVUFBQDOWHPWR4A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.545196564 (507h44m15.416279609s) [noise=true , address=77.90.31.21:4878     , piece_id=WGNCIYTKOMXU6IBC2EEVCXHVI3OAOJTRWQC5QI4463PHAHHZWYSQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: <nil>
Dec 26 19:13:15.545223244 (507h44m15.416306289s) [seg_pos={0 0}, upload=445548]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:13:15.545439286 (507h44m15.416522321s) [noise=true , address=45.91.116.5:4300     , piece_id=IZFHD6SPFWWNRPODPAF4Y755GW5YUSNSAFUV7CUOQXN36S5QBO3A, seg_pos={0 0}, upload=445548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 26 19:13:15.545453713 (507h44m15.416536788s) [noise=true , address=45.91.116.5:4300     , piece_id=IZFHD6SPFWWNRPODPAF4Y755GW5YUSNSAFUV7CUOQXN36S5QBO3A, seg_pos={0 0}, upload=445548]: Long tail cancellation due to optimal threshold
Dec 26 19:13:15.545651981 (507h44m15.416735026s) [noise=true , address=77.90.31.17:4773     , piece_id=64OJWKGO33YZWY3HJWO7KGC2DRZ346BDQ5NC2D2UNBQJZQHDZKTQ, seg_pos={0 0}, upload=445548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:51854->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:51854->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51854->77.90.31.17:4773: use of closed network connection
Dec 26 19:13:15.545659505 (507h44m15.41674254s) [noise=true , address=77.90.31.17:4773     , piece_id=64OJWKGO33YZWY3HJWO7KGC2DRZ346BDQ5NC2D2UNBQJZQHDZKTQ, seg_pos={0 0}, upload=445548]: Long tail cancellation due to optimal threshold
Dec 26 19:13:15.545872490 (507h44m15.416955545s) [noise=true , address=45.91.116.10:4083    , piece_id=PD3LCV3LGK6T6XWG73YK4H3UR3OAPGV652N5TFMQJJW7HPL6ENTA, seg_pos={0 0}, upload=445548]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF
Dec 26 19:13:15.545895023 (507h44m15.416978069s) [noise=true , address=45.91.116.10:4083    , piece_id=PD3LCV3LGK6T6XWG73YK4H3UR3OAPGV652N5TFMQJJW7HPL6ENTA, seg_pos={0 0}, upload=445548]: Long tail cancellation due to optimal threshold
Dec 26 19:13:15.545968383 (507h44m15.417051428s) [seg_pos={0 0}, upload=445548]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:13:15.546087790 (507h44m15.417170835s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:13:15.546095055 (507h44m15.4171781s) [upload=445548]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:13:15.854498168 (507h44m15.725581213s) [upload=445548]: Done uploading segments...
Dec 26 19:13:15.854517415 (507h44m15.72560045s) [upload=445548]: Upload finished. err: <nil>
Duration: 30.00331632s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..XX]
duration: 
30.00224968s
trace: 
https://jaeger.dev.storj.io/trace/449c2ef5fb79b3a4
FAIL (2025-12-26 19:12:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 26 19:12:08.879513809 (507h43m8.750596854s) [upload=445532]: Starting upload
Dec 26 19:12:08.879613609 (507h43m8.750696654s) [upload=445532]: Done starting upload
Dec 26 19:12:08.879747955 (507h43m8.75083099s) [upload=445532]: Uploading segments...
Dec 26 19:12:08.879987411 (507h43m8.751070446s) [upload=445532]: Got next segment. Inline: false
Dec 26 19:12:08.880025393 (507h43m8.751108428s) [upload=445532]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:12:08.880030743 (507h43m8.751113788s) [upload=445532]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:12:09.142061496 (507h43m9.013144541s) [seg_pos={0 0}, upload=445532]: Begin upload segment...
Dec 26 19:12:09.142127361 (507h43m9.013210406s) [seg_pos={0 0}, upload=445532]: Done begin upload segment.
Dec 26 19:12:09.142168018 (507h43m9.013251064s) [noise=true , address=145.79.227.38:4977   , piece_id=GHD5Q3774MRI47NYBTINI2RRD6RIKIB5SZRUMVHNW7NX4STADRIA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.142254013 (507h43m9.013337048s) [noise=true , address=77.90.31.14:4662     , piece_id=LQAQUIAPBBOG3EPP5H6HGW3523JPE52ZXE262ZLZ5PW5PXVAUEJA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.142289059 (507h43m9.013372105s) [noise=true , address=145.79.227.48:4301   , piece_id=42F2MWMPRULNAG7ZHURTBKEWUBU5J6JKPNUXFVLQJZLNBERTLWHA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.142937164 (507h43m9.014020199s) [noise=true , address=145.79.227.3:4150    , piece_id=FHBCE4JTYCHALFHIO6GB7CT7IDXUV2TV6IAYE4OIPFHDXYFSBQLQ, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.143158054 (507h43m9.014241089s) [noise=true , address=145.79.227.42:4122   , piece_id=S6SNYF2WJAM3MKRU3HRFE7UQPQCVHTAXSF5NWOKTTQONJQWIYSAA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.143450491 (507h43m9.014533526s) [noise=true , address=145.79.227.50:4374   , piece_id=QR5CMSGK6WSF2OU7PIX55UGLRFLINEK3SM3MNYLJVHIV2RWVKZGQ, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.143674888 (507h43m9.014757934s) [noise=true , address=145.79.227.8:4330    , piece_id=PZC3XD6RE62CXVV4CEU46OJPVZ4ML7T42GHMK2HA5AHMGW7MQFGQ, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.144001740 (507h43m9.015084786s) [noise=true , address=45.91.116.12:4225    , piece_id=LGQ5DB6HZR7V3PL6P3G4OVV5L3N65TO2YLPYHX4YCWRM6B4J25JA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.144159381 (507h43m9.015242416s) [noise=true , address=145.79.227.12:4765   , piece_id=7SJ6OM5MCHIIGTSRKOBJNCQZ4QZT2KM3RYIWAKVDWH6WY4KOK6OQ, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.144519997 (507h43m9.015603032s) [noise=true , address=145.79.227.11:4730   , piece_id=R2AUUO5NR6RWGXKFGCJSMUZ4IPDWH5KNP5N6BXHPOZWDZ623UZIA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.144541819 (507h43m9.015624854s) [noise=true , address=145.79.227.29:4942   , piece_id=FTJKOS4VK7AVVUR4DBVPZB456I67CBGV3OQZBD6ATOEZUDYOGAVA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.144854274 (507h43m9.015937319s) [noise=true , address=45.91.116.9:4263     , piece_id=WYWVEBUW4ZMVBSNJPQ7PGPJXYPMQGPKPQMWRUGAXMIQK7M33ZFIQ, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.145178101 (507h43m9.016261136s) [noise=true , address=45.91.116.5:4299     , piece_id=QD2LD5D7K6XZTW5YBI2G23GLV6LZIP4XRLJNL2EOBGOE36EFHJPA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.145236422 (507h43m9.016319467s) [noise=true , address=145.79.227.13:4803   , piece_id=5DXHYSHXCUAQCOWBSFHRXQ3IAA5WYZKPS6MRYACQHCMYFCHSUVSA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.145570658 (507h43m9.016653703s) [noise=true , address=45.91.116.7:4011     , piece_id=VOWZ36MPFHZ5MWDGMVTS55I2ZGI7EC7JQQ4AMPTPWPSLIU34NZSA, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:09.145841734 (507h43m9.016924799s) [noise=true , address=145.79.227.39:4015   , piece_id=VYK5JF6CKZZCZC2D4QDC4SO5JCE73ECUWLS4VJBGQBQSPTNKCJXQ, seg_pos={0 0}, upload=445532]: Uploading piece...
Dec 26 19:12:10.848924233 (507h43m10.720007268s) [upload=445532]: Next returned nil segment
Dec 26 19:12:10.848942408 (507h43m10.720025443s) [upload=445532]: Waiting for error group managing segments...
Dec 26 19:12:11.135489654 (507h43m11.006572689s) [noise=true , address=145.79.227.8:4330    , piece_id=PZC3XD6RE62CXVV4CEU46OJPVZ4ML7T42GHMK2HA5AHMGW7MQFGQ, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.136663099 (507h43m11.007746154s) [noise=true , address=145.79.227.38:4977   , piece_id=GHD5Q3774MRI47NYBTINI2RRD6RIKIB5SZRUMVHNW7NX4STADRIA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.144350437 (507h43m11.015433482s) [noise=true , address=145.79.227.48:4301   , piece_id=42F2MWMPRULNAG7ZHURTBKEWUBU5J6JKPNUXFVLQJZLNBERTLWHA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.155456916 (507h43m11.026539961s) [noise=true , address=145.79.227.39:4015   , piece_id=VYK5JF6CKZZCZC2D4QDC4SO5JCE73ECUWLS4VJBGQBQSPTNKCJXQ, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.166447723 (507h43m11.037530758s) [noise=true , address=145.79.227.50:4374   , piece_id=QR5CMSGK6WSF2OU7PIX55UGLRFLINEK3SM3MNYLJVHIV2RWVKZGQ, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.167063717 (507h43m11.038146762s) [noise=true , address=145.79.227.42:4122   , piece_id=S6SNYF2WJAM3MKRU3HRFE7UQPQCVHTAXSF5NWOKTTQONJQWIYSAA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.199501634 (507h43m11.070584679s) [noise=true , address=145.79.227.11:4730   , piece_id=R2AUUO5NR6RWGXKFGCJSMUZ4IPDWH5KNP5N6BXHPOZWDZ623UZIA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.389356350 (507h43m11.260439396s) [noise=true , address=45.91.116.9:4263     , piece_id=WYWVEBUW4ZMVBSNJPQ7PGPJXYPMQGPKPQMWRUGAXMIQK7M33ZFIQ, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.441577890 (507h43m11.312660935s) [noise=true , address=77.90.31.14:4662     , piece_id=LQAQUIAPBBOG3EPP5H6HGW3523JPE52ZXE262ZLZ5PW5PXVAUEJA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.490243077 (507h43m11.361326112s) [noise=true , address=145.79.227.3:4150    , piece_id=FHBCE4JTYCHALFHIO6GB7CT7IDXUV2TV6IAYE4OIPFHDXYFSBQLQ, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:11.514076611 (507h43m11.385159656s) [noise=true , address=45.91.116.5:4299     , piece_id=QD2LD5D7K6XZTW5YBI2G23GLV6LZIP4XRLJNL2EOBGOE36EFHJPA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:12.292840172 (507h43m12.163923238s) [noise=true , address=45.91.116.7:4011     , piece_id=VOWZ36MPFHZ5MWDGMVTS55I2ZGI7EC7JQQ4AMPTPWPSLIU34NZSA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:12.471914231 (507h43m12.342997276s) [noise=true , address=145.79.227.12:4765   , piece_id=7SJ6OM5MCHIIGTSRKOBJNCQZ4QZT2KM3RYIWAKVDWH6WY4KOK6OQ, seg_pos={0 0}, upload=445532]: Done uploading piece. err: <nil>
Dec 26 19:12:12.471940030 (507h43m12.343023075s) [seg_pos={0 0}, upload=445532]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:12:12.472158657 (507h43m12.343241702s) [noise=true , address=145.79.227.13:4803   , piece_id=5DXHYSHXCUAQCOWBSFHRXQ3IAA5WYZKPS6MRYACQHCMYFCHSUVSA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:39974->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:39974->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39974->145.79.227.13:4803: use of closed network connection
Dec 26 19:12:12.472166682 (507h43m12.343249727s) [noise=true , address=145.79.227.13:4803   , piece_id=5DXHYSHXCUAQCOWBSFHRXQ3IAA5WYZKPS6MRYACQHCMYFCHSUVSA, seg_pos={0 0}, upload=445532]: Long tail cancellation due to optimal threshold
Dec 26 19:12:12.472537889 (507h43m12.343620934s) [noise=true , address=45.91.116.12:4225    , piece_id=LGQ5DB6HZR7V3PL6P3G4OVV5L3N65TO2YLPYHX4YCWRM6B4J25JA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF
Dec 26 19:12:12.472554450 (507h43m12.343637485s) [noise=true , address=45.91.116.12:4225    , piece_id=LGQ5DB6HZR7V3PL6P3G4OVV5L3N65TO2YLPYHX4YCWRM6B4J25JA, seg_pos={0 0}, upload=445532]: Long tail cancellation due to optimal threshold
Dec 26 19:12:12.472563046 (507h43m12.343646101s) [noise=true , address=145.79.227.29:4942   , piece_id=FTJKOS4VK7AVVUR4DBVPZB456I67CBGV3OQZBD6ATOEZUDYOGAVA, seg_pos={0 0}, upload=445532]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Dec 26 19:12:12.472573577 (507h43m12.343656632s) [noise=true , address=145.79.227.29:4942   , piece_id=FTJKOS4VK7AVVUR4DBVPZB456I67CBGV3OQZBD6ATOEZUDYOGAVA, seg_pos={0 0}, upload=445532]: Long tail cancellation due to optimal threshold
Dec 26 19:12:12.472597772 (507h43m12.343680817s) [seg_pos={0 0}, upload=445532]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:12:12.472680851 (507h43m12.343763886s) [upload=445532]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:12:12.472687213 (507h43m12.343770258s) [upload=445532]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:12:12.761428237 (507h43m12.632511312s) [upload=445532]: Done uploading segments...
Dec 26 19:12:12.761460177 (507h43m12.632543222s) [upload=445532]: Upload finished. err: <nil>
Duration: 30.002273205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..X]
duration: 
30.00240767s
trace: 
https://jaeger.dev.storj.io/trace/6a3615d8ac2b11e9
FAIL (2025-12-26 19:11:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: I4BTXVCJULFRSP44462HB5MIF73KIKBLF2E2YCYXFDFWOAA5P66A) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: QNYH64ZHPWA6GAK5GCJPQYE75UL3FTUYIA5KROHYTHHDJ4Y7HTJA) context deadline exceeded
======LOGS======
 26 19:11:13.482421485 (507h42m13.35350452s) [upload=445519]: Starting upload
Dec 26 19:11:13.482482771 (507h42m13.353565806s) [upload=445519]: Done starting upload
Dec 26 19:11:13.482539910 (507h42m13.353622945s) [upload=445519]: Uploading segments...
Dec 26 19:11:13.482757174 (507h42m13.353840219s) [upload=445519]: Got next segment. Inline: false
Dec 26 19:11:13.482793964 (507h42m13.353876999s) [upload=445519]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:11:13.482799435 (507h42m13.35388247s) [upload=445519]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:11:13.729516237 (507h42m13.600599272s) [seg_pos={0 0}, upload=445519]: Begin upload segment...
Dec 26 19:11:13.729559750 (507h42m13.600642785s) [seg_pos={0 0}, upload=445519]: Done begin upload segment.
Dec 26 19:11:13.729591019 (507h42m13.600674064s) [noise=true , address=145.79.227.34:4905   , piece_id=7D6H2NK3SPTGX6HGG7MAKV6OX4Q2W7NANDQX4HRHA6FNGKFULZ4A, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.729595508 (507h42m13.600678553s) [noise=true , address=145.79.227.42:4121   , piece_id=X76EHFOJMKFLRXJSL4HIHUXEALIEZXIK4ACOIARS46SX5AAAJJ2Q, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.730166075 (507h42m13.60124911s) [noise=true , address=145.79.227.3:4150    , piece_id=L2NKX32W54VVIJUCH2PHOPKILQ4H4ABGYG2LIWCYY7U6DFIAJU7A, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.730453752 (507h42m13.601536787s) [noise=true , address=45.91.116.5:4297     , piece_id=VSG4HNSPJEVDIBRO62UDYXZDRRKTVVIBGKTNJEH4HHMOJLNMGIIQ, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.730759435 (507h42m13.60184247s) [noise=true , address=145.79.227.49:4339   , piece_id=6DYQ7LAVN4JNGOIXBNDY6XDJYRD2HHLJCROIIROYHSC7XDIUPE5A, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.731047733 (507h42m13.602130768s) [noise=true , address=145.79.227.36:4518   , piece_id=HZVUME2DASHUUKPM2NSFUFV2IKWNFU2WS752R3DLYXTBZBJ2IPLQ, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.731343467 (507h42m13.602426502s) [noise=true , address=145.79.227.50:4373   , piece_id=WC3S23HYQXMN4PLRYT5ECUMYKXSE3KMA6NZTSH5QFNOIHWF7VDVA, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.731594735 (507h42m13.60267778s) [noise=true , address=45.91.116.7:4011     , piece_id=NO5FOJHSUGIGGTIGSZH6HLYE23KKSBHSPDZLDRRPPKRPGDLIMEYA, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.731842968 (507h42m13.602926003s) [noise=true , address=145.79.227.30:4762   , piece_id=WQHF4MYYEKLFGFUA3I2OUB66SICMO25C5NB3PHIJW7CCM7WKLPYQ, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.732092613 (507h42m13.603175648s) [noise=true , address=145.79.227.16:4909   , piece_id=CC4SIVC6WZU5ZBCSDK42RLODW64ECQAVUPC2SGES6COXWBGU6QGA, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.732357558 (507h42m13.603440593s) [noise=true , address=145.79.227.39:4013   , piece_id=CWDF3UFRRE4Y4LINPGROYDRIV5PZGDZ3UWYNWMLBJY2ADUGICOWQ, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.732612433 (507h42m13.603695468s) [noise=true , address=77.90.31.14:4662     , piece_id=I4BTXVCJULFRSP44462HB5MIF73KIKBLF2E2YCYXFDFWOAA5P66A, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.732872378 (507h42m13.603955423s) [noise=true , address=145.79.227.51:4409   , piece_id=FRZ7CYX4ZY7EPPR67XMO3HHODH7QMAMMLJUSXQR3JUT4QJW7VXDQ, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.733149165 (507h42m13.6042322s) [noise=true , address=45.91.116.6:4975     , piece_id=QNYH64ZHPWA6GAK5GCJPQYE75UL3FTUYIA5KROHYTHHDJ4Y7HTJA, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.733413608 (507h42m13.604496643s) [noise=true , address=145.79.227.8:4330    , piece_id=KKEFM4WM4CW3JWDXBWF72VTP4QZ7K7XZLGBQ7V7JC6WJV7JI6LGA, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:13.733651331 (507h42m13.604734366s) [noise=true , address=145.79.227.12:4767   , piece_id=OGASQ7RSSWKR25M4S4NAHXDXKFSA52VBGRUZJO7LXNDJRT5TGEEQ, seg_pos={0 0}, upload=445519]: Uploading piece...
Dec 26 19:11:15.448746123 (507h42m15.319829168s) [upload=445519]: Next returned nil segment
Dec 26 19:11:15.448759358 (507h42m15.319842393s) [upload=445519]: Waiting for error group managing segments...
Dec 26 19:11:15.731970902 (507h42m15.603053937s) [noise=true , address=145.79.227.3:4150    , piece_id=L2NKX32W54VVIJUCH2PHOPKILQ4H4ABGYG2LIWCYY7U6DFIAJU7A, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:15.735176166 (507h42m15.606259211s) [noise=true , address=145.79.227.8:4330    , piece_id=KKEFM4WM4CW3JWDXBWF72VTP4QZ7K7XZLGBQ7V7JC6WJV7JI6LGA, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:15.735643436 (507h42m15.606726471s) [noise=true , address=145.79.227.42:4121   , piece_id=X76EHFOJMKFLRXJSL4HIHUXEALIEZXIK4ACOIARS46SX5AAAJJ2Q, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:15.743699345 (507h42m15.61478238s) [noise=true , address=145.79.227.39:4013   , piece_id=CWDF3UFRRE4Y4LINPGROYDRIV5PZGDZ3UWYNWMLBJY2ADUGICOWQ, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:15.746296742 (507h42m15.617379787s) [noise=true , address=145.79.227.50:4373   , piece_id=WC3S23HYQXMN4PLRYT5ECUMYKXSE3KMA6NZTSH5QFNOIHWF7VDVA, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:15.755983056 (507h42m15.627066091s) [noise=true , address=145.79.227.36:4518   , piece_id=HZVUME2DASHUUKPM2NSFUFV2IKWNFU2WS752R3DLYXTBZBJ2IPLQ, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:15.776272360 (507h42m15.647355395s) [noise=true , address=145.79.227.49:4339   , piece_id=6DYQ7LAVN4JNGOIXBNDY6XDJYRD2HHLJCROIIROYHSC7XDIUPE5A, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:16.051080213 (507h42m15.922163248s) [noise=true , address=145.79.227.12:4767   , piece_id=OGASQ7RSSWKR25M4S4NAHXDXKFSA52VBGRUZJO7LXNDJRT5TGEEQ, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:16.071263345 (507h42m15.94234638s) [noise=true , address=45.91.116.7:4011     , piece_id=NO5FOJHSUGIGGTIGSZH6HLYE23KKSBHSPDZLDRRPPKRPGDLIMEYA, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:16.119159858 (507h42m15.990242903s) [noise=true , address=45.91.116.5:4297     , piece_id=VSG4HNSPJEVDIBRO62UDYXZDRRKTVVIBGKTNJEH4HHMOJLNMGIIQ, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:16.654836421 (507h42m16.525919456s) [noise=true , address=77.90.31.14:4662     , piece_id=I4BTXVCJULFRSP44462HB5MIF73KIKBLF2E2YCYXFDFWOAA5P66A, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:17.024857737 (507h42m16.895940772s) [noise=true , address=145.79.227.30:4762   , piece_id=WQHF4MYYEKLFGFUA3I2OUB66SICMO25C5NB3PHIJW7CCM7WKLPYQ, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:17.375008655 (507h42m17.2460917s) [noise=true , address=45.91.116.6:4975     , piece_id=QNYH64ZHPWA6GAK5GCJPQYE75UL3FTUYIA5KROHYTHHDJ4Y7HTJA, seg_pos={0 0}, upload=445519]: Done uploading piece. err: <nil>
Dec 26 19:11:17.375029545 (507h42m17.24611259s) [seg_pos={0 0}, upload=445519]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:11:17.375237922 (507h42m17.246320967s) [noise=true , address=145.79.227.51:4409   , piece_id=FRZ7CYX4ZY7EPPR67XMO3HHODH7QMAMMLJUSXQR3JUT4QJW7VXDQ, seg_pos={0 0}, upload=445519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF
Dec 26 19:11:17.375245997 (507h42m17.246329042s) [noise=true , address=145.79.227.51:4409   , piece_id=FRZ7CYX4ZY7EPPR67XMO3HHODH7QMAMMLJUSXQR3JUT4QJW7VXDQ, seg_pos={0 0}, upload=445519]: Long tail cancellation due to optimal threshold
Dec 26 19:11:17.375282667 (507h42m17.246365702s) [noise=true , address=145.79.227.16:4909   , piece_id=CC4SIVC6WZU5ZBCSDK42RLODW64ECQAVUPC2SGES6COXWBGU6QGA, seg_pos={0 0}, upload=445519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x): protocol: write tcp 188.166.225.170:54008->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:54008->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54008->145.79.227.16:4909: use of closed network connection
Dec 26 19:11:17.375291183 (507h42m17.246374219s) [noise=true , address=145.79.227.16:4909   , piece_id=CC4SIVC6WZU5ZBCSDK42RLODW64ECQAVUPC2SGES6COXWBGU6QGA, seg_pos={0 0}, upload=445519]: Long tail cancellation due to optimal threshold
Dec 26 19:11:17.375457600 (507h42m17.246540635s) [noise=true , address=145.79.227.34:4905   , piece_id=7D6H2NK3SPTGX6HGG7MAKV6OX4Q2W7NANDQX4HRHA6FNGKFULZ4A, seg_pos={0 0}, upload=445519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF
Dec 26 19:11:17.375464904 (507h42m17.246547949s) [noise=true , address=145.79.227.34:4905   , piece_id=7D6H2NK3SPTGX6HGG7MAKV6OX4Q2W7NANDQX4HRHA6FNGKFULZ4A, seg_pos={0 0}, upload=445519]: Long tail cancellation due to optimal threshold
Dec 26 19:11:17.375483579 (507h42m17.246566614s) [seg_pos={0 0}, upload=445519]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:11:17.375545417 (507h42m17.246628452s) [upload=445519]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:11:17.375551118 (507h42m17.246634153s) [upload=445519]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:11:17.680502108 (507h42m17.551585153s) [upload=445519]: Done uploading segments...
Dec 26 19:11:17.680524301 (507h42m17.551607346s) [upload=445519]: Upload finished. err: <nil>
Duration: 30.002429423s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X..]
duration: 
5.866938061s
trace: 
https://jaeger.dev.storj.io/trace/4fb242c1ca63cde9
FAIL (2025-12-26 19:10:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
76]: Uploading segments...
Dec 26 19:07:58.829719377 (507h38m58.700802423s) [upload=445476]: Got next segment. Inline: false
Dec 26 19:07:58.829886045 (507h38m58.70096908s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:07:58.829893289 (507h38m58.700976334s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:07:59.078125286 (507h38m58.949208321s) [seg_pos={0 0}, upload=445476]: Begin upload segment...
Dec 26 19:07:59.078167226 (507h38m58.949250271s) [seg_pos={0 0}, upload=445476]: Done begin upload segment.
Dec 26 19:07:59.078191192 (507h38m58.949274227s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.078213474 (507h38m58.949296509s) [noise=true , address=145.79.227.45:4230   , piece_id=MOAF5UFG5HV25RALW7UC6F4VYBRQB6ZOWHXNSA4ZYH22TRFZ3DHA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.078804158 (507h38m58.949887194s) [noise=true , address=145.79.227.39:4015   , piece_id=DPNIORZQVXFIHZ2K5TWGG5KE2CKQQPI3WNOUQGTIXB2AHK337F2A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079361931 (507h38m58.950444976s) [noise=true , address=45.91.116.6:4973     , piece_id=HS2M3JWSGTYUHCL6XZZ24R5QNUPEE6M2GJEPAVT6Q26QFEKQVWSA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079368925 (507h38m58.95045197s) [noise=true , address=145.79.227.21:4475   , piece_id=SDQQVBOTIQMWDXOAU5T62IFC5ZJIPAVFTH3VYOZKXXXCNMVX54RQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079712688 (507h38m58.950795733s) [noise=true , address=77.90.31.16:4736     , piece_id=YUBBVQGFHXKJJFQ4MREG57P34XAFWXJDXE2D2M22SYUNJTQML73A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079760069 (507h38m58.950843104s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080055451 (507h38m58.951138496s) [noise=true , address=145.79.227.36:4519   , piece_id=RX62UEQATE435TNVOUZ3IXXL2BNNUNTEO6YNNOHM22DFFPJ6RFEA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080079858 (507h38m58.951162893s) [noise=true , address=145.79.227.8:4331    , piece_id=V6WPLY6J7VO7DDG6PWKKBECQRH56WYYFDACYRYPBE63YP5SSSGYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080329423 (507h38m58.951412468s) [noise=true , address=145.79.227.12:4765   , piece_id=KO35ISTU674PGNYCMTA7NJTUBXB7HNR63KR3O6G5Q5YOY4GUGTYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080362175 (507h38m58.951445221s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080603735 (507h38m58.951686781s) [noise=true , address=45.91.116.14:4189    , piece_id=ZD3IHJOGF2SSFB7ARYD4UFUO3M5NRO4MTW6TJIMLG24SZ37B3XZA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080641167 (507h38m58.951724202s) [noise=true , address=145.79.227.48:4301   , piece_id=XT4BPC6VODMM5FTELIZK7HQL3F54UIXUOFGCYSTBYV7WNT4YPLVA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080910610 (507h38m58.951993645s) [noise=true , address=45.91.116.12:4225    , piece_id=GOJH35OWCFHABKODHCGPOHKTF6Q7QHFQHFRJXD4CNUSK4IQXGUIQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.081398770 (507h38m58.952481815s) [noise=true , address=145.79.227.11:4732   , piece_id=LZGUKUXAZC2GO4R7YTIBFTDCWXIRQZCJ2YGLJGHHVG4YO3XUUZBQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.082393122 (507h38m58.953476157s) [noise=true , address=145.79.227.37:4566   , piece_id=BQ5QP2AM5IZTX5A4ZP7MN56VQNFVC2K4CBMZ4QUBRNWJOQQ47YYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:08:00.795180572 (507h39m0.666263617s) [upload=445476]: Next returned nil segment
Dec 26 19:08:00.795206471 (507h39m0.666289516s) [upload=445476]: Waiting for error group managing segments...
Dec 26 19:08:01.081456711 (507h39m0.952539756s) [noise=true , address=145.79.227.21:4475   , piece_id=SDQQVBOTIQMWDXOAU5T62IFC5ZJIPAVFTH3VYOZKXXXCNMVX54RQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.082523122 (507h39m0.953606177s) [noise=true , address=145.79.227.45:4230   , piece_id=MOAF5UFG5HV25RALW7UC6F4VYBRQB6ZOWHXNSA4ZYH22TRFZ3DHA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.083249766 (507h39m0.954332801s) [noise=true , address=145.79.227.37:4566   , piece_id=BQ5QP2AM5IZTX5A4ZP7MN56VQNFVC2K4CBMZ4QUBRNWJOQQ47YYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.086432396 (507h39m0.957515441s) [noise=true , address=145.79.227.11:4732   , piece_id=LZGUKUXAZC2GO4R7YTIBFTDCWXIRQZCJ2YGLJGHHVG4YO3XUUZBQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.087747770 (507h39m0.958830816s) [noise=true , address=145.79.227.36:4519   , piece_id=RX62UEQATE435TNVOUZ3IXXL2BNNUNTEO6YNNOHM22DFFPJ6RFEA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.088174363 (507h39m0.959257408s) [noise=true , address=145.79.227.48:4301   , piece_id=XT4BPC6VODMM5FTELIZK7HQL3F54UIXUOFGCYSTBYV7WNT4YPLVA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.097603417 (507h39m0.968686452s) [noise=true , address=145.79.227.39:4015   , piece_id=DPNIORZQVXFIHZ2K5TWGG5KE2CKQQPI3WNOUQGTIXB2AHK337F2A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.098327586 (507h39m0.969410631s) [noise=true , address=145.79.227.8:4331    , piece_id=V6WPLY6J7VO7DDG6PWKKBECQRH56WYYFDACYRYPBE63YP5SSSGYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.262890443 (507h39m1.133973488s) [noise=true , address=77.90.31.16:4736     , piece_id=YUBBVQGFHXKJJFQ4MREG57P34XAFWXJDXE2D2M22SYUNJTQML73A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.560519471 (507h39m1.431602516s) [noise=true , address=145.79.227.12:4765   , piece_id=KO35ISTU674PGNYCMTA7NJTUBXB7HNR63KR3O6G5Q5YOY4GUGTYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.653258552 (507h39m1.524341607s) [noise=true , address=45.91.116.14:4189    , piece_id=ZD3IHJOGF2SSFB7ARYD4UFUO3M5NRO4MTW6TJIMLG24SZ37B3XZA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.054582283 (507h39m1.925665328s) [noise=true , address=45.91.116.6:4973     , piece_id=HS2M3JWSGTYUHCL6XZZ24R5QNUPEE6M2GJEPAVT6Q26QFEKQVWSA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.136148379 (507h39m2.007231424s) [noise=true , address=45.91.116.12:4225    , piece_id=GOJH35OWCFHABKODHCGPOHKTF6Q7QHFQHFRJXD4CNUSK4IQXGUIQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.136175902 (507h39m2.007258947s) [seg_pos={0 0}, upload=445476]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:08:02.136405379 (507h39m2.007488424s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection
Dec 26 19:08:02.136415037 (507h39m2.007498092s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136427160 (507h39m2.007510205s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection
Dec 26 19:08:02.136438693 (507h39m2.007521758s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136610480 (507h39m2.007693545s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 26 19:08:02.136619977 (507h39m2.007703023s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136707213 (507h39m2.007790258s) [seg_pos={0 0}, upload=445476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:08:02.136798147 (507h39m2.007881192s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:08:02.136804438 (507h39m2.007887484s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:08:02.439624010 (507h39m2.310707055s) [upload=445476]: Done uploading segments...
Dec 26 19:08:02.439648597 (507h39m2.310731632s) [upload=445476]: Upload finished. err: <nil>
Duration: 5.866988756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [.X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X.]
duration: 
20.704529215s
trace: 
https://jaeger.dev.storj.io/trace/4fb242c1ca63cde9
FAIL (2025-12-26 19:09:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
76]: Uploading segments...
Dec 26 19:07:58.829719377 (507h38m58.700802423s) [upload=445476]: Got next segment. Inline: false
Dec 26 19:07:58.829886045 (507h38m58.70096908s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:07:58.829893289 (507h38m58.700976334s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:07:59.078125286 (507h38m58.949208321s) [seg_pos={0 0}, upload=445476]: Begin upload segment...
Dec 26 19:07:59.078167226 (507h38m58.949250271s) [seg_pos={0 0}, upload=445476]: Done begin upload segment.
Dec 26 19:07:59.078191192 (507h38m58.949274227s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.078213474 (507h38m58.949296509s) [noise=true , address=145.79.227.45:4230   , piece_id=MOAF5UFG5HV25RALW7UC6F4VYBRQB6ZOWHXNSA4ZYH22TRFZ3DHA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.078804158 (507h38m58.949887194s) [noise=true , address=145.79.227.39:4015   , piece_id=DPNIORZQVXFIHZ2K5TWGG5KE2CKQQPI3WNOUQGTIXB2AHK337F2A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079361931 (507h38m58.950444976s) [noise=true , address=45.91.116.6:4973     , piece_id=HS2M3JWSGTYUHCL6XZZ24R5QNUPEE6M2GJEPAVT6Q26QFEKQVWSA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079368925 (507h38m58.95045197s) [noise=true , address=145.79.227.21:4475   , piece_id=SDQQVBOTIQMWDXOAU5T62IFC5ZJIPAVFTH3VYOZKXXXCNMVX54RQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079712688 (507h38m58.950795733s) [noise=true , address=77.90.31.16:4736     , piece_id=YUBBVQGFHXKJJFQ4MREG57P34XAFWXJDXE2D2M22SYUNJTQML73A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079760069 (507h38m58.950843104s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080055451 (507h38m58.951138496s) [noise=true , address=145.79.227.36:4519   , piece_id=RX62UEQATE435TNVOUZ3IXXL2BNNUNTEO6YNNOHM22DFFPJ6RFEA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080079858 (507h38m58.951162893s) [noise=true , address=145.79.227.8:4331    , piece_id=V6WPLY6J7VO7DDG6PWKKBECQRH56WYYFDACYRYPBE63YP5SSSGYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080329423 (507h38m58.951412468s) [noise=true , address=145.79.227.12:4765   , piece_id=KO35ISTU674PGNYCMTA7NJTUBXB7HNR63KR3O6G5Q5YOY4GUGTYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080362175 (507h38m58.951445221s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080603735 (507h38m58.951686781s) [noise=true , address=45.91.116.14:4189    , piece_id=ZD3IHJOGF2SSFB7ARYD4UFUO3M5NRO4MTW6TJIMLG24SZ37B3XZA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080641167 (507h38m58.951724202s) [noise=true , address=145.79.227.48:4301   , piece_id=XT4BPC6VODMM5FTELIZK7HQL3F54UIXUOFGCYSTBYV7WNT4YPLVA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080910610 (507h38m58.951993645s) [noise=true , address=45.91.116.12:4225    , piece_id=GOJH35OWCFHABKODHCGPOHKTF6Q7QHFQHFRJXD4CNUSK4IQXGUIQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.081398770 (507h38m58.952481815s) [noise=true , address=145.79.227.11:4732   , piece_id=LZGUKUXAZC2GO4R7YTIBFTDCWXIRQZCJ2YGLJGHHVG4YO3XUUZBQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.082393122 (507h38m58.953476157s) [noise=true , address=145.79.227.37:4566   , piece_id=BQ5QP2AM5IZTX5A4ZP7MN56VQNFVC2K4CBMZ4QUBRNWJOQQ47YYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:08:00.795180572 (507h39m0.666263617s) [upload=445476]: Next returned nil segment
Dec 26 19:08:00.795206471 (507h39m0.666289516s) [upload=445476]: Waiting for error group managing segments...
Dec 26 19:08:01.081456711 (507h39m0.952539756s) [noise=true , address=145.79.227.21:4475   , piece_id=SDQQVBOTIQMWDXOAU5T62IFC5ZJIPAVFTH3VYOZKXXXCNMVX54RQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.082523122 (507h39m0.953606177s) [noise=true , address=145.79.227.45:4230   , piece_id=MOAF5UFG5HV25RALW7UC6F4VYBRQB6ZOWHXNSA4ZYH22TRFZ3DHA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.083249766 (507h39m0.954332801s) [noise=true , address=145.79.227.37:4566   , piece_id=BQ5QP2AM5IZTX5A4ZP7MN56VQNFVC2K4CBMZ4QUBRNWJOQQ47YYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.086432396 (507h39m0.957515441s) [noise=true , address=145.79.227.11:4732   , piece_id=LZGUKUXAZC2GO4R7YTIBFTDCWXIRQZCJ2YGLJGHHVG4YO3XUUZBQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.087747770 (507h39m0.958830816s) [noise=true , address=145.79.227.36:4519   , piece_id=RX62UEQATE435TNVOUZ3IXXL2BNNUNTEO6YNNOHM22DFFPJ6RFEA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.088174363 (507h39m0.959257408s) [noise=true , address=145.79.227.48:4301   , piece_id=XT4BPC6VODMM5FTELIZK7HQL3F54UIXUOFGCYSTBYV7WNT4YPLVA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.097603417 (507h39m0.968686452s) [noise=true , address=145.79.227.39:4015   , piece_id=DPNIORZQVXFIHZ2K5TWGG5KE2CKQQPI3WNOUQGTIXB2AHK337F2A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.098327586 (507h39m0.969410631s) [noise=true , address=145.79.227.8:4331    , piece_id=V6WPLY6J7VO7DDG6PWKKBECQRH56WYYFDACYRYPBE63YP5SSSGYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.262890443 (507h39m1.133973488s) [noise=true , address=77.90.31.16:4736     , piece_id=YUBBVQGFHXKJJFQ4MREG57P34XAFWXJDXE2D2M22SYUNJTQML73A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.560519471 (507h39m1.431602516s) [noise=true , address=145.79.227.12:4765   , piece_id=KO35ISTU674PGNYCMTA7NJTUBXB7HNR63KR3O6G5Q5YOY4GUGTYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.653258552 (507h39m1.524341607s) [noise=true , address=45.91.116.14:4189    , piece_id=ZD3IHJOGF2SSFB7ARYD4UFUO3M5NRO4MTW6TJIMLG24SZ37B3XZA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.054582283 (507h39m1.925665328s) [noise=true , address=45.91.116.6:4973     , piece_id=HS2M3JWSGTYUHCL6XZZ24R5QNUPEE6M2GJEPAVT6Q26QFEKQVWSA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.136148379 (507h39m2.007231424s) [noise=true , address=45.91.116.12:4225    , piece_id=GOJH35OWCFHABKODHCGPOHKTF6Q7QHFQHFRJXD4CNUSK4IQXGUIQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.136175902 (507h39m2.007258947s) [seg_pos={0 0}, upload=445476]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:08:02.136405379 (507h39m2.007488424s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection
Dec 26 19:08:02.136415037 (507h39m2.007498092s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136427160 (507h39m2.007510205s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection
Dec 26 19:08:02.136438693 (507h39m2.007521758s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136610480 (507h39m2.007693545s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 26 19:08:02.136619977 (507h39m2.007703023s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136707213 (507h39m2.007790258s) [seg_pos={0 0}, upload=445476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:08:02.136798147 (507h39m2.007881192s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:08:02.136804438 (507h39m2.007887484s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:08:02.439624010 (507h39m2.310707055s) [upload=445476]: Done uploading segments...
Dec 26 19:08:02.439648597 (507h39m2.310731632s) [upload=445476]: Upload finished. err: <nil>
Duration: 20.704559903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [..X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..X]
duration: 
30.003103333s
trace: 
https://jaeger.dev.storj.io/trace/4fb242c1ca63cde9
FAIL (2025-12-26 19:08:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
76]: Uploading segments...
Dec 26 19:07:58.829719377 (507h38m58.700802423s) [upload=445476]: Got next segment. Inline: false
Dec 26 19:07:58.829886045 (507h38m58.70096908s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 19:07:58.829893289 (507h38m58.700976334s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:07:59.078125286 (507h38m58.949208321s) [seg_pos={0 0}, upload=445476]: Begin upload segment...
Dec 26 19:07:59.078167226 (507h38m58.949250271s) [seg_pos={0 0}, upload=445476]: Done begin upload segment.
Dec 26 19:07:59.078191192 (507h38m58.949274227s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.078213474 (507h38m58.949296509s) [noise=true , address=145.79.227.45:4230   , piece_id=MOAF5UFG5HV25RALW7UC6F4VYBRQB6ZOWHXNSA4ZYH22TRFZ3DHA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.078804158 (507h38m58.949887194s) [noise=true , address=145.79.227.39:4015   , piece_id=DPNIORZQVXFIHZ2K5TWGG5KE2CKQQPI3WNOUQGTIXB2AHK337F2A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079361931 (507h38m58.950444976s) [noise=true , address=45.91.116.6:4973     , piece_id=HS2M3JWSGTYUHCL6XZZ24R5QNUPEE6M2GJEPAVT6Q26QFEKQVWSA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079368925 (507h38m58.95045197s) [noise=true , address=145.79.227.21:4475   , piece_id=SDQQVBOTIQMWDXOAU5T62IFC5ZJIPAVFTH3VYOZKXXXCNMVX54RQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079712688 (507h38m58.950795733s) [noise=true , address=77.90.31.16:4736     , piece_id=YUBBVQGFHXKJJFQ4MREG57P34XAFWXJDXE2D2M22SYUNJTQML73A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.079760069 (507h38m58.950843104s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080055451 (507h38m58.951138496s) [noise=true , address=145.79.227.36:4519   , piece_id=RX62UEQATE435TNVOUZ3IXXL2BNNUNTEO6YNNOHM22DFFPJ6RFEA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080079858 (507h38m58.951162893s) [noise=true , address=145.79.227.8:4331    , piece_id=V6WPLY6J7VO7DDG6PWKKBECQRH56WYYFDACYRYPBE63YP5SSSGYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080329423 (507h38m58.951412468s) [noise=true , address=145.79.227.12:4765   , piece_id=KO35ISTU674PGNYCMTA7NJTUBXB7HNR63KR3O6G5Q5YOY4GUGTYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080362175 (507h38m58.951445221s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080603735 (507h38m58.951686781s) [noise=true , address=45.91.116.14:4189    , piece_id=ZD3IHJOGF2SSFB7ARYD4UFUO3M5NRO4MTW6TJIMLG24SZ37B3XZA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080641167 (507h38m58.951724202s) [noise=true , address=145.79.227.48:4301   , piece_id=XT4BPC6VODMM5FTELIZK7HQL3F54UIXUOFGCYSTBYV7WNT4YPLVA, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.080910610 (507h38m58.951993645s) [noise=true , address=45.91.116.12:4225    , piece_id=GOJH35OWCFHABKODHCGPOHKTF6Q7QHFQHFRJXD4CNUSK4IQXGUIQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.081398770 (507h38m58.952481815s) [noise=true , address=145.79.227.11:4732   , piece_id=LZGUKUXAZC2GO4R7YTIBFTDCWXIRQZCJ2YGLJGHHVG4YO3XUUZBQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:07:59.082393122 (507h38m58.953476157s) [noise=true , address=145.79.227.37:4566   , piece_id=BQ5QP2AM5IZTX5A4ZP7MN56VQNFVC2K4CBMZ4QUBRNWJOQQ47YYQ, seg_pos={0 0}, upload=445476]: Uploading piece...
Dec 26 19:08:00.795180572 (507h39m0.666263617s) [upload=445476]: Next returned nil segment
Dec 26 19:08:00.795206471 (507h39m0.666289516s) [upload=445476]: Waiting for error group managing segments...
Dec 26 19:08:01.081456711 (507h39m0.952539756s) [noise=true , address=145.79.227.21:4475   , piece_id=SDQQVBOTIQMWDXOAU5T62IFC5ZJIPAVFTH3VYOZKXXXCNMVX54RQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.082523122 (507h39m0.953606177s) [noise=true , address=145.79.227.45:4230   , piece_id=MOAF5UFG5HV25RALW7UC6F4VYBRQB6ZOWHXNSA4ZYH22TRFZ3DHA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.083249766 (507h39m0.954332801s) [noise=true , address=145.79.227.37:4566   , piece_id=BQ5QP2AM5IZTX5A4ZP7MN56VQNFVC2K4CBMZ4QUBRNWJOQQ47YYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.086432396 (507h39m0.957515441s) [noise=true , address=145.79.227.11:4732   , piece_id=LZGUKUXAZC2GO4R7YTIBFTDCWXIRQZCJ2YGLJGHHVG4YO3XUUZBQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.087747770 (507h39m0.958830816s) [noise=true , address=145.79.227.36:4519   , piece_id=RX62UEQATE435TNVOUZ3IXXL2BNNUNTEO6YNNOHM22DFFPJ6RFEA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.088174363 (507h39m0.959257408s) [noise=true , address=145.79.227.48:4301   , piece_id=XT4BPC6VODMM5FTELIZK7HQL3F54UIXUOFGCYSTBYV7WNT4YPLVA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.097603417 (507h39m0.968686452s) [noise=true , address=145.79.227.39:4015   , piece_id=DPNIORZQVXFIHZ2K5TWGG5KE2CKQQPI3WNOUQGTIXB2AHK337F2A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.098327586 (507h39m0.969410631s) [noise=true , address=145.79.227.8:4331    , piece_id=V6WPLY6J7VO7DDG6PWKKBECQRH56WYYFDACYRYPBE63YP5SSSGYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.262890443 (507h39m1.133973488s) [noise=true , address=77.90.31.16:4736     , piece_id=YUBBVQGFHXKJJFQ4MREG57P34XAFWXJDXE2D2M22SYUNJTQML73A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.560519471 (507h39m1.431602516s) [noise=true , address=145.79.227.12:4765   , piece_id=KO35ISTU674PGNYCMTA7NJTUBXB7HNR63KR3O6G5Q5YOY4GUGTYQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:01.653258552 (507h39m1.524341607s) [noise=true , address=45.91.116.14:4189    , piece_id=ZD3IHJOGF2SSFB7ARYD4UFUO3M5NRO4MTW6TJIMLG24SZ37B3XZA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.054582283 (507h39m1.925665328s) [noise=true , address=45.91.116.6:4973     , piece_id=HS2M3JWSGTYUHCL6XZZ24R5QNUPEE6M2GJEPAVT6Q26QFEKQVWSA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.136148379 (507h39m2.007231424s) [noise=true , address=45.91.116.12:4225    , piece_id=GOJH35OWCFHABKODHCGPOHKTF6Q7QHFQHFRJXD4CNUSK4IQXGUIQ, seg_pos={0 0}, upload=445476]: Done uploading piece. err: <nil>
Dec 26 19:08:02.136175902 (507h39m2.007258947s) [seg_pos={0 0}, upload=445476]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:08:02.136405379 (507h39m2.007488424s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H): protocol: write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection; write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39942->145.79.227.27:4690: use of closed network connection
Dec 26 19:08:02.136415037 (507h39m2.007498092s) [noise=true , address=145.79.227.27:4690   , piece_id=N3FL5OIEIXUZK35XJA27K7PKPU3YZ3IOIQG5SRQV2FQVP7VU4HMA, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136427160 (507h39m2.007510205s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58854->145.79.227.19:4078: use of closed network connection
Dec 26 19:08:02.136438693 (507h39m2.007521758s) [noise=true , address=145.79.227.19:4078   , piece_id=NI2GASNZZSL6HH252VZE56BHHR4AZXFMGJPYZXKJ4ONGBFTEPKKA, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136610480 (507h39m2.007693545s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 26 19:08:02.136619977 (507h39m2.007703023s) [noise=true , address=45.91.116.7:4010     , piece_id=HYQRCGFRJRSTN4RPW5SAQHRN4CD37Z7TOEJOCD5FNMJ6L2ZSBN2A, seg_pos={0 0}, upload=445476]: Long tail cancellation due to optimal threshold
Dec 26 19:08:02.136707213 (507h39m2.007790258s) [seg_pos={0 0}, upload=445476]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:08:02.136798147 (507h39m2.007881192s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:08:02.136804438 (507h39m2.007887484s) [upload=445476]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:08:02.439624010 (507h39m2.310707055s) [upload=445476]: Done uploading segments...
Dec 26 19:08:02.439648597 (507h39m2.310731632s) [upload=445476]: Upload finished. err: <nil>
Duration: 30.003114756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [...X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X..]
duration: 
26.003278003s
trace: 
https://jaeger.dev.storj.io/trace/586c7491033fb10a
FAIL (2025-12-26 19:07:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ) context deadline exceeded
======LOGS======
0.113594039 (507h36m29.984677084s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:05:30.362004785 (507h36m30.233087831s) [seg_pos={0 0}, upload=445436]: Begin upload segment...
Dec 26 19:05:30.362061444 (507h36m30.233144489s) [seg_pos={0 0}, upload=445436]: Done begin upload segment.
Dec 26 19:05:30.362098524 (507h36m30.233181569s) [noise=true , address=145.79.227.39:4014   , piece_id=D4T6KNJJWHHSQQRGWH5KDTHKVT5M234IVELFCDILNKQGTRBOB3YQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362190630 (507h36m30.233273675s) [noise=true , address=45.91.116.9:4263     , piece_id=RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362869542 (507h36m30.233952577s) [noise=true , address=145.79.227.31:4799   , piece_id=KRXC5W2F4QJAH3KIPQ7V3K5WMFWSNOS564ICBBFHOYZ777IDQEGA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362960676 (507h36m30.234043711s) [noise=true , address=145.79.227.46:4601   , piece_id=NSHVBR4ISPN33DADS7VOMOZ45TCXULA6TSOC6UJDKROVZXVCZJRA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.363333385 (507h36m30.23441642s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.363698842 (507h36m30.234781877s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.364049279 (507h36m30.235132324s) [noise=true , address=77.90.31.19:4844     , piece_id=RZ72XV4OEFHGS6KNSTK7NRPLGODUOE76MIHNPNB7BCBQ62RVFQUQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.364415335 (507h36m30.23549837s) [noise=true , address=145.79.227.11:4731   , piece_id=D5FLCMQYE76JISUCTK2QK3XTIKDYOMQDNWMITIYPKFU3YWF7YJRA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.365545849 (507h36m30.236628894s) [noise=true , address=145.79.227.52:4445   , piece_id=WDIWYXXQPORX2VXNSQF3I27QTPSZQV6R5X7BFKQGCERFPTXVVTKA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.366550512 (507h36m30.237633547s) [noise=true , address=145.79.227.8:4330    , piece_id=BFYK66BXNFR6SMGQCFILI6H4SDV6RKKGQRNUIW5BOLRNM2AH5ELA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.367815640 (507h36m30.238898675s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.369590791 (507h36m30.240673826s) [noise=true , address=45.91.116.7:4009     , piece_id=PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.370994954 (507h36m30.242077989s) [noise=true , address=145.79.227.51:4411   , piece_id=XP3NUVHHMISOGIXNCDWSKDZS2EWRX74URAXIFMQ77I3TWREADQHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.371454650 (507h36m30.242537685s) [noise=true , address=145.79.227.49:4337   , piece_id=2HIAJFC6XFCFUX6DJM6VWDB37VOJWPNEW5VSDSHUCBYXBDJXFQOA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.371788886 (507h36m30.242871921s) [noise=true , address=145.79.227.35:4482   , piece_id=TOFT4LMPNW3S2Q6UT5JNFZOC2MZHNUO6SZ4UIDBQPUECWV6SFYHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.372334095 (507h36m30.24341713s) [noise=true , address=145.79.227.42:4121   , piece_id=2H6QZ7BPYIUONWGOUWAXQL7HBSXAZG7RPKED6LP6FJF2PVFOOQQA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:32.084085378 (507h36m31.955168413s) [upload=445436]: Next returned nil segment
Dec 26 19:05:32.084103694 (507h36m31.955186739s) [upload=445436]: Waiting for error group managing segments...
Dec 26 19:05:32.370489964 (507h36m32.241573009s) [noise=true , address=145.79.227.31:4799   , piece_id=KRXC5W2F4QJAH3KIPQ7V3K5WMFWSNOS564ICBBFHOYZ777IDQEGA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.373769108 (507h36m32.244852163s) [noise=true , address=145.79.227.42:4121   , piece_id=2H6QZ7BPYIUONWGOUWAXQL7HBSXAZG7RPKED6LP6FJF2PVFOOQQA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.375551822 (507h36m32.246634857s) [noise=true , address=145.79.227.46:4601   , piece_id=NSHVBR4ISPN33DADS7VOMOZ45TCXULA6TSOC6UJDKROVZXVCZJRA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.379558882 (507h36m32.250641917s) [noise=true , address=145.79.227.51:4411   , piece_id=XP3NUVHHMISOGIXNCDWSKDZS2EWRX74URAXIFMQ77I3TWREADQHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.391847452 (507h36m32.262930487s) [noise=true , address=145.79.227.11:4731   , piece_id=D5FLCMQYE76JISUCTK2QK3XTIKDYOMQDNWMITIYPKFU3YWF7YJRA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.392003579 (507h36m32.263086614s) [noise=true , address=145.79.227.39:4014   , piece_id=D4T6KNJJWHHSQQRGWH5KDTHKVT5M234IVELFCDILNKQGTRBOB3YQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.392543777 (507h36m32.263626823s) [noise=true , address=145.79.227.52:4445   , piece_id=WDIWYXXQPORX2VXNSQF3I27QTPSZQV6R5X7BFKQGCERFPTXVVTKA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.395485209 (507h36m32.266568244s) [noise=true , address=145.79.227.8:4330    , piece_id=BFYK66BXNFR6SMGQCFILI6H4SDV6RKKGQRNUIW5BOLRNM2AH5ELA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.395638460 (507h36m32.266721505s) [noise=true , address=145.79.227.35:4482   , piece_id=TOFT4LMPNW3S2Q6UT5JNFZOC2MZHNUO6SZ4UIDBQPUECWV6SFYHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.400900370 (507h36m32.271983415s) [noise=true , address=145.79.227.49:4337   , piece_id=2HIAJFC6XFCFUX6DJM6VWDB37VOJWPNEW5VSDSHUCBYXBDJXFQOA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.574248726 (507h36m32.445331761s) [noise=true , address=77.90.31.19:4844     , piece_id=RZ72XV4OEFHGS6KNSTK7NRPLGODUOE76MIHNPNB7BCBQ62RVFQUQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.597878942 (507h36m32.468961977s) [noise=true , address=45.91.116.9:4263     , piece_id=RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.823422828 (507h36m32.694505863s) [noise=true , address=45.91.116.7:4009     , piece_id=PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.823446061 (507h36m32.694529096s) [seg_pos={0 0}, upload=445436]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:05:32.823645271 (507h36m32.694728306s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection
Dec 26 19:05:32.823651543 (507h36m32.694734578s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.823789836 (507h36m32.694872871s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection
Dec 26 19:05:32.823811588 (507h36m32.694894633s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.823963537 (507h36m32.695046572s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection
Dec 26 19:05:32.823972604 (507h36m32.695055639s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.824019193 (507h36m32.695102228s) [seg_pos={0 0}, upload=445436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:05:32.824089317 (507h36m32.695172352s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:05:32.824094366 (507h36m32.695177411s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:05:33.130819775 (507h36m33.00190284s) [upload=445436]: Done uploading segments...
Dec 26 19:05:33.130859881 (507h36m33.001942926s) [upload=445436]: Upload finished. err: <nil>
Duration: 26.003315485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X.]
duration: 
6.341510391s
trace: 
https://jaeger.dev.storj.io/trace/586c7491033fb10a
FAIL (2025-12-26 19:06:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ) context deadline exceeded
======LOGS======
0.113594039 (507h36m29.984677084s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:05:30.362004785 (507h36m30.233087831s) [seg_pos={0 0}, upload=445436]: Begin upload segment...
Dec 26 19:05:30.362061444 (507h36m30.233144489s) [seg_pos={0 0}, upload=445436]: Done begin upload segment.
Dec 26 19:05:30.362098524 (507h36m30.233181569s) [noise=true , address=145.79.227.39:4014   , piece_id=D4T6KNJJWHHSQQRGWH5KDTHKVT5M234IVELFCDILNKQGTRBOB3YQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362190630 (507h36m30.233273675s) [noise=true , address=45.91.116.9:4263     , piece_id=RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362869542 (507h36m30.233952577s) [noise=true , address=145.79.227.31:4799   , piece_id=KRXC5W2F4QJAH3KIPQ7V3K5WMFWSNOS564ICBBFHOYZ777IDQEGA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362960676 (507h36m30.234043711s) [noise=true , address=145.79.227.46:4601   , piece_id=NSHVBR4ISPN33DADS7VOMOZ45TCXULA6TSOC6UJDKROVZXVCZJRA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.363333385 (507h36m30.23441642s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.363698842 (507h36m30.234781877s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.364049279 (507h36m30.235132324s) [noise=true , address=77.90.31.19:4844     , piece_id=RZ72XV4OEFHGS6KNSTK7NRPLGODUOE76MIHNPNB7BCBQ62RVFQUQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.364415335 (507h36m30.23549837s) [noise=true , address=145.79.227.11:4731   , piece_id=D5FLCMQYE76JISUCTK2QK3XTIKDYOMQDNWMITIYPKFU3YWF7YJRA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.365545849 (507h36m30.236628894s) [noise=true , address=145.79.227.52:4445   , piece_id=WDIWYXXQPORX2VXNSQF3I27QTPSZQV6R5X7BFKQGCERFPTXVVTKA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.366550512 (507h36m30.237633547s) [noise=true , address=145.79.227.8:4330    , piece_id=BFYK66BXNFR6SMGQCFILI6H4SDV6RKKGQRNUIW5BOLRNM2AH5ELA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.367815640 (507h36m30.238898675s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.369590791 (507h36m30.240673826s) [noise=true , address=45.91.116.7:4009     , piece_id=PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.370994954 (507h36m30.242077989s) [noise=true , address=145.79.227.51:4411   , piece_id=XP3NUVHHMISOGIXNCDWSKDZS2EWRX74URAXIFMQ77I3TWREADQHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.371454650 (507h36m30.242537685s) [noise=true , address=145.79.227.49:4337   , piece_id=2HIAJFC6XFCFUX6DJM6VWDB37VOJWPNEW5VSDSHUCBYXBDJXFQOA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.371788886 (507h36m30.242871921s) [noise=true , address=145.79.227.35:4482   , piece_id=TOFT4LMPNW3S2Q6UT5JNFZOC2MZHNUO6SZ4UIDBQPUECWV6SFYHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.372334095 (507h36m30.24341713s) [noise=true , address=145.79.227.42:4121   , piece_id=2H6QZ7BPYIUONWGOUWAXQL7HBSXAZG7RPKED6LP6FJF2PVFOOQQA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:32.084085378 (507h36m31.955168413s) [upload=445436]: Next returned nil segment
Dec 26 19:05:32.084103694 (507h36m31.955186739s) [upload=445436]: Waiting for error group managing segments...
Dec 26 19:05:32.370489964 (507h36m32.241573009s) [noise=true , address=145.79.227.31:4799   , piece_id=KRXC5W2F4QJAH3KIPQ7V3K5WMFWSNOS564ICBBFHOYZ777IDQEGA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.373769108 (507h36m32.244852163s) [noise=true , address=145.79.227.42:4121   , piece_id=2H6QZ7BPYIUONWGOUWAXQL7HBSXAZG7RPKED6LP6FJF2PVFOOQQA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.375551822 (507h36m32.246634857s) [noise=true , address=145.79.227.46:4601   , piece_id=NSHVBR4ISPN33DADS7VOMOZ45TCXULA6TSOC6UJDKROVZXVCZJRA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.379558882 (507h36m32.250641917s) [noise=true , address=145.79.227.51:4411   , piece_id=XP3NUVHHMISOGIXNCDWSKDZS2EWRX74URAXIFMQ77I3TWREADQHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.391847452 (507h36m32.262930487s) [noise=true , address=145.79.227.11:4731   , piece_id=D5FLCMQYE76JISUCTK2QK3XTIKDYOMQDNWMITIYPKFU3YWF7YJRA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.392003579 (507h36m32.263086614s) [noise=true , address=145.79.227.39:4014   , piece_id=D4T6KNJJWHHSQQRGWH5KDTHKVT5M234IVELFCDILNKQGTRBOB3YQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.392543777 (507h36m32.263626823s) [noise=true , address=145.79.227.52:4445   , piece_id=WDIWYXXQPORX2VXNSQF3I27QTPSZQV6R5X7BFKQGCERFPTXVVTKA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.395485209 (507h36m32.266568244s) [noise=true , address=145.79.227.8:4330    , piece_id=BFYK66BXNFR6SMGQCFILI6H4SDV6RKKGQRNUIW5BOLRNM2AH5ELA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.395638460 (507h36m32.266721505s) [noise=true , address=145.79.227.35:4482   , piece_id=TOFT4LMPNW3S2Q6UT5JNFZOC2MZHNUO6SZ4UIDBQPUECWV6SFYHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.400900370 (507h36m32.271983415s) [noise=true , address=145.79.227.49:4337   , piece_id=2HIAJFC6XFCFUX6DJM6VWDB37VOJWPNEW5VSDSHUCBYXBDJXFQOA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.574248726 (507h36m32.445331761s) [noise=true , address=77.90.31.19:4844     , piece_id=RZ72XV4OEFHGS6KNSTK7NRPLGODUOE76MIHNPNB7BCBQ62RVFQUQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.597878942 (507h36m32.468961977s) [noise=true , address=45.91.116.9:4263     , piece_id=RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.823422828 (507h36m32.694505863s) [noise=true , address=45.91.116.7:4009     , piece_id=PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.823446061 (507h36m32.694529096s) [seg_pos={0 0}, upload=445436]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:05:32.823645271 (507h36m32.694728306s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection
Dec 26 19:05:32.823651543 (507h36m32.694734578s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.823789836 (507h36m32.694872871s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection
Dec 26 19:05:32.823811588 (507h36m32.694894633s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.823963537 (507h36m32.695046572s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection
Dec 26 19:05:32.823972604 (507h36m32.695055639s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.824019193 (507h36m32.695102228s) [seg_pos={0 0}, upload=445436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:05:32.824089317 (507h36m32.695172352s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:05:32.824094366 (507h36m32.695177411s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:05:33.130819775 (507h36m33.00190284s) [upload=445436]: Done uploading segments...
Dec 26 19:05:33.130859881 (507h36m33.001942926s) [upload=445436]: Upload finished. err: <nil>
Duration: 6.341534156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......X]
duration: 
30.002887874s
trace: 
https://jaeger.dev.storj.io/trace/586c7491033fb10a
FAIL (2025-12-26 19:05:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ) context deadline exceeded
======LOGS======
0.113594039 (507h36m29.984677084s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 19:05:30.362004785 (507h36m30.233087831s) [seg_pos={0 0}, upload=445436]: Begin upload segment...
Dec 26 19:05:30.362061444 (507h36m30.233144489s) [seg_pos={0 0}, upload=445436]: Done begin upload segment.
Dec 26 19:05:30.362098524 (507h36m30.233181569s) [noise=true , address=145.79.227.39:4014   , piece_id=D4T6KNJJWHHSQQRGWH5KDTHKVT5M234IVELFCDILNKQGTRBOB3YQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362190630 (507h36m30.233273675s) [noise=true , address=45.91.116.9:4263     , piece_id=RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362869542 (507h36m30.233952577s) [noise=true , address=145.79.227.31:4799   , piece_id=KRXC5W2F4QJAH3KIPQ7V3K5WMFWSNOS564ICBBFHOYZ777IDQEGA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.362960676 (507h36m30.234043711s) [noise=true , address=145.79.227.46:4601   , piece_id=NSHVBR4ISPN33DADS7VOMOZ45TCXULA6TSOC6UJDKROVZXVCZJRA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.363333385 (507h36m30.23441642s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.363698842 (507h36m30.234781877s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.364049279 (507h36m30.235132324s) [noise=true , address=77.90.31.19:4844     , piece_id=RZ72XV4OEFHGS6KNSTK7NRPLGODUOE76MIHNPNB7BCBQ62RVFQUQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.364415335 (507h36m30.23549837s) [noise=true , address=145.79.227.11:4731   , piece_id=D5FLCMQYE76JISUCTK2QK3XTIKDYOMQDNWMITIYPKFU3YWF7YJRA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.365545849 (507h36m30.236628894s) [noise=true , address=145.79.227.52:4445   , piece_id=WDIWYXXQPORX2VXNSQF3I27QTPSZQV6R5X7BFKQGCERFPTXVVTKA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.366550512 (507h36m30.237633547s) [noise=true , address=145.79.227.8:4330    , piece_id=BFYK66BXNFR6SMGQCFILI6H4SDV6RKKGQRNUIW5BOLRNM2AH5ELA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.367815640 (507h36m30.238898675s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.369590791 (507h36m30.240673826s) [noise=true , address=45.91.116.7:4009     , piece_id=PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.370994954 (507h36m30.242077989s) [noise=true , address=145.79.227.51:4411   , piece_id=XP3NUVHHMISOGIXNCDWSKDZS2EWRX74URAXIFMQ77I3TWREADQHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.371454650 (507h36m30.242537685s) [noise=true , address=145.79.227.49:4337   , piece_id=2HIAJFC6XFCFUX6DJM6VWDB37VOJWPNEW5VSDSHUCBYXBDJXFQOA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.371788886 (507h36m30.242871921s) [noise=true , address=145.79.227.35:4482   , piece_id=TOFT4LMPNW3S2Q6UT5JNFZOC2MZHNUO6SZ4UIDBQPUECWV6SFYHQ, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:30.372334095 (507h36m30.24341713s) [noise=true , address=145.79.227.42:4121   , piece_id=2H6QZ7BPYIUONWGOUWAXQL7HBSXAZG7RPKED6LP6FJF2PVFOOQQA, seg_pos={0 0}, upload=445436]: Uploading piece...
Dec 26 19:05:32.084085378 (507h36m31.955168413s) [upload=445436]: Next returned nil segment
Dec 26 19:05:32.084103694 (507h36m31.955186739s) [upload=445436]: Waiting for error group managing segments...
Dec 26 19:05:32.370489964 (507h36m32.241573009s) [noise=true , address=145.79.227.31:4799   , piece_id=KRXC5W2F4QJAH3KIPQ7V3K5WMFWSNOS564ICBBFHOYZ777IDQEGA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.373769108 (507h36m32.244852163s) [noise=true , address=145.79.227.42:4121   , piece_id=2H6QZ7BPYIUONWGOUWAXQL7HBSXAZG7RPKED6LP6FJF2PVFOOQQA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.375551822 (507h36m32.246634857s) [noise=true , address=145.79.227.46:4601   , piece_id=NSHVBR4ISPN33DADS7VOMOZ45TCXULA6TSOC6UJDKROVZXVCZJRA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.379558882 (507h36m32.250641917s) [noise=true , address=145.79.227.51:4411   , piece_id=XP3NUVHHMISOGIXNCDWSKDZS2EWRX74URAXIFMQ77I3TWREADQHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.391847452 (507h36m32.262930487s) [noise=true , address=145.79.227.11:4731   , piece_id=D5FLCMQYE76JISUCTK2QK3XTIKDYOMQDNWMITIYPKFU3YWF7YJRA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.392003579 (507h36m32.263086614s) [noise=true , address=145.79.227.39:4014   , piece_id=D4T6KNJJWHHSQQRGWH5KDTHKVT5M234IVELFCDILNKQGTRBOB3YQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.392543777 (507h36m32.263626823s) [noise=true , address=145.79.227.52:4445   , piece_id=WDIWYXXQPORX2VXNSQF3I27QTPSZQV6R5X7BFKQGCERFPTXVVTKA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.395485209 (507h36m32.266568244s) [noise=true , address=145.79.227.8:4330    , piece_id=BFYK66BXNFR6SMGQCFILI6H4SDV6RKKGQRNUIW5BOLRNM2AH5ELA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.395638460 (507h36m32.266721505s) [noise=true , address=145.79.227.35:4482   , piece_id=TOFT4LMPNW3S2Q6UT5JNFZOC2MZHNUO6SZ4UIDBQPUECWV6SFYHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.400900370 (507h36m32.271983415s) [noise=true , address=145.79.227.49:4337   , piece_id=2HIAJFC6XFCFUX6DJM6VWDB37VOJWPNEW5VSDSHUCBYXBDJXFQOA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.574248726 (507h36m32.445331761s) [noise=true , address=77.90.31.19:4844     , piece_id=RZ72XV4OEFHGS6KNSTK7NRPLGODUOE76MIHNPNB7BCBQ62RVFQUQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.597878942 (507h36m32.468961977s) [noise=true , address=45.91.116.9:4263     , piece_id=RGR2BS2UL4S5BJE5H2R7O6S45AFOIGKERNLZZFBNP36C7HUX6RNQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.823422828 (507h36m32.694505863s) [noise=true , address=45.91.116.7:4009     , piece_id=PU2VJQSLF3BHW2QVDGD44762MNL5QOKMZIALNCSD5PICRVCOKLKQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: <nil>
Dec 26 19:05:32.823446061 (507h36m32.694529096s) [seg_pos={0 0}, upload=445436]: Segment reached optimal threshold of 13 pieces.
Dec 26 19:05:32.823645271 (507h36m32.694728306s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34558->77.90.31.16:4736: use of closed network connection
Dec 26 19:05:32.823651543 (507h36m32.694734578s) [noise=true , address=77.90.31.16:4736     , piece_id=QH7PF5TIXCUDGGMV624OUZZQWAH7IIKOP366NAAYUL6LD5PH6X3Q, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.823789836 (507h36m32.694872871s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35908->45.91.116.8:4046: use of closed network connection
Dec 26 19:05:32.823811588 (507h36m32.694894633s) [noise=true , address=45.91.116.8:4046     , piece_id=FBOA2P72RPE2TLYA6XX5JMPDKSWZL27FK4XOPG4FK2UMWCW6XOAA, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.823963537 (507h36m32.695046572s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection; write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37316->45.91.116.4:4937: use of closed network connection
Dec 26 19:05:32.823972604 (507h36m32.695055639s) [noise=true , address=45.91.116.4:4937     , piece_id=KRGKD3T2V6QZGYRYORBZIITNKINFB7KWGCTX3VXI2J73IXBKDIHQ, seg_pos={0 0}, upload=445436]: Long tail cancellation due to optimal threshold
Dec 26 19:05:32.824019193 (507h36m32.695102228s) [seg_pos={0 0}, upload=445436]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 19:05:32.824089317 (507h36m32.695172352s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 19:05:32.824094366 (507h36m32.695177411s) [upload=445436]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:05:33.130819775 (507h36m33.00190284s) [upload=445436]: Done uploading segments...
Dec 26 19:05:33.130859881 (507h36m33.001942926s) [upload=445436]: Upload finished. err: <nil>
Duration: 30.002909955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.......]
duration: 
6.571068931s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:04:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 6.571093647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X......]
duration: 
6.641363413s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:03:51 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 6.641392298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [..X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.....]
duration: 
6.139824134s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:03:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 6.139852368s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [...X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X....]
duration: 
26.061205085s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:02:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 26.061234832s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [....X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X...]
duration: 
7.009042179s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:02:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 7.009068809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.....X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X..]
duration: 
5.954730574s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:01:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 5.954787372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [......X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X.]
duration: 
5.763140673s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:00:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 5.763171241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [.......X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...X]
duration: 
30.002486829s
trace: 
https://jaeger.dev.storj.io/trace/7737bed5cb5ba8f9
FAIL (2025-12-26 19:00:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
s...
Dec 26 18:59:56.390795721 (507h30m56.261878757s) [upload=445352]: Got next segment. Inline: false
Dec 26 18:59:56.390827472 (507h30m56.261910507s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 26 18:59:56.390831551 (507h30m56.261914596s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:59:56.644272206 (507h30m56.515355241s) [seg_pos={0 0}, upload=445352]: Begin upload segment...
Dec 26 18:59:56.644329075 (507h30m56.51541212s) [seg_pos={0 0}, upload=445352]: Done begin upload segment.
Dec 26 18:59:56.644381233 (507h30m56.515464268s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644409477 (507h30m56.515492512s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.644946479 (507h30m56.516029515s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645303550 (507h30m56.516386595s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645648116 (507h30m56.516731151s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.645953267 (507h30m56.517036302s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646310488 (507h30m56.517393523s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646606190 (507h30m56.517689225s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.646997856 (507h30m56.518080891s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.648086650 (507h30m56.519169685s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.649474802 (507h30m56.520557837s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.651947942 (507h30m56.523030987s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.653424584 (507h30m56.524507629s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.654541941 (507h30m56.525624986s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.655670951 (507h30m56.526753986s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:56.656910101 (507h30m56.527993146s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Uploading piece...
Dec 26 18:59:58.368361174 (507h30m58.239444209s) [upload=445352]: Next returned nil segment
Dec 26 18:59:58.368375050 (507h30m58.239458096s) [upload=445352]: Waiting for error group managing segments...
Dec 26 18:59:58.649576898 (507h30m58.520659933s) [noise=true , address=145.79.227.52:4447   , piece_id=ZBNQVBSOMUUGBWE5TJJSW5RINHAYHHTBV3DEGPNLS3GUHP2LKRJA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.658435466 (507h30m58.529518501s) [noise=true , address=145.79.227.29:4942   , piece_id=MNJ7NVBKQ6TX6RDTRBQAEAVIIZHSJSGXBIPHUUKCVMHOVFJSN4PQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.663856499 (507h30m58.534939544s) [noise=true , address=145.79.227.22:4509   , piece_id=SPULGCFDC72AYLNLGV7HFJJFZMR42IBG63A7Q6IMCM4SUAOQDJ3Q, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.664735132 (507h30m58.535818167s) [noise=true , address=145.79.227.8:4331    , piece_id=HAEM2BUXFHLJQ6UZJQONV4UIXLEG65YT4THEYXZKCRGKCDGQMC4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.667277222 (507h30m58.538360257s) [noise=true , address=145.79.227.36:4518   , piece_id=WPUGP45QVC2HDOOK3DVYWEAN67AJYDFS2HAVYMNYX3SUA5ZYQ4PA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.675808528 (507h30m58.546891563s) [noise=true , address=145.79.227.11:4732   , piece_id=HRLDXNXMIHXPKJ2TTZ3KRPV2FQKDT3WVCP25D5FKCESAXYTTSITA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.677430045 (507h30m58.54851309s) [noise=true , address=145.79.227.48:4303   , piece_id=XTDV6G6ASQGUH6WHJBKL5X64W23RKBUOHHMNSQICPXYRR6D2D3UA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.896817538 (507h30m58.767900573s) [noise=true , address=45.91.116.10:4082    , piece_id=DA4VJXUND7IEEYDEMKEZ7ITEANFGDL5WLDCNU3ZHS3V6E46CGMIQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:58.934177198 (507h30m58.805260233s) [noise=true , address=45.91.116.12:4228    , piece_id=KPS5KU2JG7K4NDJ3GR4U4XYO223D6MOADYGHMZQHBU44YB2CDEQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.028212437 (507h30m58.899295472s) [noise=true , address=45.91.116.6:4973     , piece_id=MB7LNXHFG2AQUFYNGP5QJEBAVSHI26UOZKL3JFQVIFF4U2Q6XXGA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.599148820 (507h30m59.470231865s) [noise=true , address=45.91.116.11:4117    , piece_id=QX7XJDVKU5AJTJYE7ASQ2R3P4L67RYYS6JHS4UNWU75MDHRSQ7VQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.664405230 (507h30m59.535488275s) [noise=true , address=45.91.116.9:4264     , piece_id=INBF5B3ZRP7N5KHGRRIPAYFJOBUO4QNJ6PHUT5XXBABP5VTL5W4A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750385191 (507h30m59.621468236s) [noise=true , address=45.91.116.5:4297     , piece_id=DPXR37YZPLWGZU4BRB66RAV67KVMFBILXFTCARVLWM2SN6CUG4YA, seg_pos={0 0}, upload=445352]: Done uploading piece. err: <nil>
Dec 26 18:59:59.750409818 (507h30m59.621492853s) [seg_pos={0 0}, upload=445352]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:59:59.750607635 (507h30m59.62169068s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54712->145.79.227.51:4409: use of closed network connection
Dec 26 18:59:59.750617042 (507h30m59.621700078s) [noise=true , address=145.79.227.51:4409   , piece_id=B772HMMZRUYJZYC5DCBWVV6R75KBCZKTMNNMHP4SL6SKU3TMT35A, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.750839295 (507h30m59.62192236s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 26 18:59:59.750856889 (507h30m59.621939924s) [noise=true , address=145.79.227.35:4483   , piece_id=XIWQRZ6KQKJNMKQAZXYYMZQH57LAVSJFZRM55QKUU7FL5BOQRRQQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751049005 (507h30m59.62213204s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36364->145.79.227.6:4257: use of closed network connection
Dec 26 18:59:59.751058073 (507h30m59.622141108s) [noise=true , address=145.79.227.6:4257    , piece_id=EPDQO4LODRWBL2X46K4HB2TSVTDWE6VRXQPILHMNRQ2J2BK3D6TQ, seg_pos={0 0}, upload=445352]: Long tail cancellation due to optimal threshold
Dec 26 18:59:59.751083831 (507h30m59.622166876s) [seg_pos={0 0}, upload=445352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:59:59.751170096 (507h30m59.622253141s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:59:59.751175526 (507h30m59.622258561s) [upload=445352]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 19:00:00.043669415 (507h30m59.91475246s) [upload=445352]: Done uploading segments...
Dec 26 19:00:00.043690887 (507h30m59.914773922s) [upload=445352]: Upload finished. err: <nil>
Duration: 30.002498261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [........X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X...]
duration: 
25.745031246s
trace: 
https://jaeger.dev.storj.io/trace/2378689e60deb6aa
FAIL (2025-12-26 18:59:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ) context deadline exceeded
======LOGS======
87 (507h28m13.950489922s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:57:14.334959965 (507h28m14.20604301s) [seg_pos={0 0}, upload=445310]: Begin upload segment...
Dec 26 18:57:14.335029998 (507h28m14.206113043s) [seg_pos={0 0}, upload=445310]: Done begin upload segment.
Dec 26 18:57:14.335069604 (507h28m14.206152649s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.336443991 (507h28m14.207527046s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337251549 (507h28m14.208334594s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337747332 (507h28m14.208830378s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338176410 (507h28m14.209259445s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338517389 (507h28m14.209600424s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339032390 (507h28m14.210115435s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339405180 (507h28m14.210488215s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340145910 (507h28m14.211228945s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340668946 (507h28m14.211751981s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341047276 (507h28m14.212130311s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341445184 (507h28m14.212528229s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341944655 (507h28m14.21302769s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342044546 (507h28m14.213127581s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342587178 (507h28m14.213670223s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342938708 (507h28m14.214021743s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:16.025263441 (507h28m15.896346476s) [upload=445310]: Next returned nil segment
Dec 26 18:57:16.025275343 (507h28m15.896358388s) [upload=445310]: Waiting for error group managing segments...
Dec 26 18:57:16.341572919 (507h28m16.212655964s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.356248083 (507h28m16.227331138s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.357319534 (507h28m16.228402579s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.360048570 (507h28m16.231131605s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.366955922 (507h28m16.238038957s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.367287835 (507h28m16.23837087s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.498672535 (507h28m16.36975558s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.540594571 (507h28m16.411677626s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.596235571 (507h28m16.467318606s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.657285911 (507h28m16.528368957s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.706833209 (507h28m16.577916254s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.822845286 (507h28m16.693928351s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194874415 (507h28m17.06595745s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194899042 (507h28m17.065982077s) [seg_pos={0 0}, upload=445310]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:57:17.195194714 (507h28m17.066277749s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection
Dec 26 18:57:17.195203200 (507h28m17.066286245s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195257284 (507h28m17.066340319s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection
Dec 26 18:57:17.195271601 (507h28m17.066354636s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195368315 (507h28m17.06645136s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection
Dec 26 18:57:17.195374056 (507h28m17.066457091s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195389205 (507h28m17.06647224s) [seg_pos={0 0}, upload=445310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:57:17.195465821 (507h28m17.066548866s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:57:17.195471041 (507h28m17.066554076s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 18:57:17.496284783 (507h28m17.367367839s) [upload=445310]: Done uploading segments...
Dec 26 18:57:17.496304901 (507h28m17.367387936s) [upload=445310]: Upload finished. err: <nil>
Duration: 25.745060462s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
18 out of 60 is failed (threshold=3) [.........X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X..]
duration: 
6.158649902s
trace: 
https://jaeger.dev.storj.io/trace/2378689e60deb6aa
FAIL (2025-12-26 18:59:14 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ) context deadline exceeded
======LOGS======
87 (507h28m13.950489922s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:57:14.334959965 (507h28m14.20604301s) [seg_pos={0 0}, upload=445310]: Begin upload segment...
Dec 26 18:57:14.335029998 (507h28m14.206113043s) [seg_pos={0 0}, upload=445310]: Done begin upload segment.
Dec 26 18:57:14.335069604 (507h28m14.206152649s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.336443991 (507h28m14.207527046s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337251549 (507h28m14.208334594s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337747332 (507h28m14.208830378s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338176410 (507h28m14.209259445s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338517389 (507h28m14.209600424s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339032390 (507h28m14.210115435s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339405180 (507h28m14.210488215s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340145910 (507h28m14.211228945s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340668946 (507h28m14.211751981s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341047276 (507h28m14.212130311s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341445184 (507h28m14.212528229s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341944655 (507h28m14.21302769s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342044546 (507h28m14.213127581s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342587178 (507h28m14.213670223s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342938708 (507h28m14.214021743s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:16.025263441 (507h28m15.896346476s) [upload=445310]: Next returned nil segment
Dec 26 18:57:16.025275343 (507h28m15.896358388s) [upload=445310]: Waiting for error group managing segments...
Dec 26 18:57:16.341572919 (507h28m16.212655964s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.356248083 (507h28m16.227331138s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.357319534 (507h28m16.228402579s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.360048570 (507h28m16.231131605s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.366955922 (507h28m16.238038957s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.367287835 (507h28m16.23837087s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.498672535 (507h28m16.36975558s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.540594571 (507h28m16.411677626s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.596235571 (507h28m16.467318606s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.657285911 (507h28m16.528368957s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.706833209 (507h28m16.577916254s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.822845286 (507h28m16.693928351s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194874415 (507h28m17.06595745s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194899042 (507h28m17.065982077s) [seg_pos={0 0}, upload=445310]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:57:17.195194714 (507h28m17.066277749s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection
Dec 26 18:57:17.195203200 (507h28m17.066286245s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195257284 (507h28m17.066340319s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection
Dec 26 18:57:17.195271601 (507h28m17.066354636s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195368315 (507h28m17.06645136s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection
Dec 26 18:57:17.195374056 (507h28m17.066457091s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195389205 (507h28m17.06647224s) [seg_pos={0 0}, upload=445310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:57:17.195465821 (507h28m17.066548866s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:57:17.195471041 (507h28m17.066554076s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 18:57:17.496284783 (507h28m17.367367839s) [upload=445310]: Done uploading segments...
Dec 26 18:57:17.496304901 (507h28m17.367387936s) [upload=445310]: Upload finished. err: <nil>
Duration: 6.158677294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
19 out of 60 is failed (threshold=3) [X.........X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X.]
duration: 
6.214579813s
trace: 
https://jaeger.dev.storj.io/trace/2378689e60deb6aa
FAIL (2025-12-26 18:58:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ) context deadline exceeded
======LOGS======
87 (507h28m13.950489922s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:57:14.334959965 (507h28m14.20604301s) [seg_pos={0 0}, upload=445310]: Begin upload segment...
Dec 26 18:57:14.335029998 (507h28m14.206113043s) [seg_pos={0 0}, upload=445310]: Done begin upload segment.
Dec 26 18:57:14.335069604 (507h28m14.206152649s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.336443991 (507h28m14.207527046s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337251549 (507h28m14.208334594s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337747332 (507h28m14.208830378s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338176410 (507h28m14.209259445s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338517389 (507h28m14.209600424s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339032390 (507h28m14.210115435s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339405180 (507h28m14.210488215s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340145910 (507h28m14.211228945s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340668946 (507h28m14.211751981s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341047276 (507h28m14.212130311s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341445184 (507h28m14.212528229s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341944655 (507h28m14.21302769s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342044546 (507h28m14.213127581s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342587178 (507h28m14.213670223s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342938708 (507h28m14.214021743s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:16.025263441 (507h28m15.896346476s) [upload=445310]: Next returned nil segment
Dec 26 18:57:16.025275343 (507h28m15.896358388s) [upload=445310]: Waiting for error group managing segments...
Dec 26 18:57:16.341572919 (507h28m16.212655964s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.356248083 (507h28m16.227331138s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.357319534 (507h28m16.228402579s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.360048570 (507h28m16.231131605s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.366955922 (507h28m16.238038957s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.367287835 (507h28m16.23837087s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.498672535 (507h28m16.36975558s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.540594571 (507h28m16.411677626s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.596235571 (507h28m16.467318606s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.657285911 (507h28m16.528368957s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.706833209 (507h28m16.577916254s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.822845286 (507h28m16.693928351s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194874415 (507h28m17.06595745s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194899042 (507h28m17.065982077s) [seg_pos={0 0}, upload=445310]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:57:17.195194714 (507h28m17.066277749s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection
Dec 26 18:57:17.195203200 (507h28m17.066286245s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195257284 (507h28m17.066340319s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection
Dec 26 18:57:17.195271601 (507h28m17.066354636s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195368315 (507h28m17.06645136s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection
Dec 26 18:57:17.195374056 (507h28m17.066457091s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195389205 (507h28m17.06647224s) [seg_pos={0 0}, upload=445310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:57:17.195465821 (507h28m17.066548866s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:57:17.195471041 (507h28m17.066554076s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 18:57:17.496284783 (507h28m17.367367839s) [upload=445310]: Done uploading segments...
Dec 26 18:57:17.496304901 (507h28m17.367387936s) [upload=445310]: Upload finished. err: <nil>
Duration: 6.214604352s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
20 out of 60 is failed (threshold=3) [XX.........X.....X..X..X.X.X.....XXX.X...X.XXX..X...X...X..X]
duration: 
30.002780519s
trace: 
https://jaeger.dev.storj.io/trace/2378689e60deb6aa
FAIL (2025-12-26 18:57:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ) context deadline exceeded
======LOGS======
87 (507h28m13.950489922s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 26 18:57:14.334959965 (507h28m14.20604301s) [seg_pos={0 0}, upload=445310]: Begin upload segment...
Dec 26 18:57:14.335029998 (507h28m14.206113043s) [seg_pos={0 0}, upload=445310]: Done begin upload segment.
Dec 26 18:57:14.335069604 (507h28m14.206152649s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.336443991 (507h28m14.207527046s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337251549 (507h28m14.208334594s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.337747332 (507h28m14.208830378s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338176410 (507h28m14.209259445s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.338517389 (507h28m14.209600424s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339032390 (507h28m14.210115435s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.339405180 (507h28m14.210488215s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340145910 (507h28m14.211228945s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.340668946 (507h28m14.211751981s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341047276 (507h28m14.212130311s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341445184 (507h28m14.212528229s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.341944655 (507h28m14.21302769s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342044546 (507h28m14.213127581s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342587178 (507h28m14.213670223s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:14.342938708 (507h28m14.214021743s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Uploading piece...
Dec 26 18:57:16.025263441 (507h28m15.896346476s) [upload=445310]: Next returned nil segment
Dec 26 18:57:16.025275343 (507h28m15.896358388s) [upload=445310]: Waiting for error group managing segments...
Dec 26 18:57:16.341572919 (507h28m16.212655964s) [noise=true , address=145.79.227.49:4338   , piece_id=7BUVVYZY62KZ2MNVK6DBDR3ST3OM3XTOBNGVXFFHGRTLFMBL4RYA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.356248083 (507h28m16.227331138s) [noise=true , address=145.79.227.46:4601   , piece_id=QV3LEV3WPUYT6PCNUO4XA3BRZQA4VJNGSSY3FRFEAHDRYOAYA4TA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.357319534 (507h28m16.228402579s) [noise=true , address=145.79.227.3:4150    , piece_id=HFYMS3JJBVBKQN322U644ZUH7SQA6QK6RU3STNNVC6O4V6AJFRGQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.360048570 (507h28m16.231131605s) [noise=true , address=145.79.227.50:4375   , piece_id=UHWWVWCHFEPNITDXMNY6POCWAKYQALVJ7BLWHRIPQGHLJ4U7YQSA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.366955922 (507h28m16.238038957s) [noise=true , address=145.79.227.37:4567   , piece_id=IFFI6KDMATSDGPJJE5SGXAGAFYUCNEQCAWVA3P2QS53563BWEAEQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.367287835 (507h28m16.23837087s) [noise=true , address=145.79.227.27:4690   , piece_id=LDZDLMDRJD3YBSVHWPVA7HY6Y3JRMDVHO7D4IA3EF35M7HBNWTBA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.498672535 (507h28m16.36975558s) [noise=true , address=145.79.227.12:4767   , piece_id=RKPMTWJQMMXHTRVPKX5VDQ3KONCOED4QUCGJRSHYJVL36FPEWPHQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.540594571 (507h28m16.411677626s) [noise=true , address=77.90.31.14:4664     , piece_id=56ILLMCWWROCN7NL6HQ55KFCPJUZHQHVK2NXI3R75DDCM22Y46LQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.596235571 (507h28m16.467318606s) [noise=true , address=77.90.31.10:4520     , piece_id=HFC6DQZ55UOIZNWA4KABFP2WYPIEYUN72GLP5ONWNJ3XNILJI5GQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.657285911 (507h28m16.528368957s) [noise=true , address=45.91.116.10:4082    , piece_id=BNZJYMHVP3WZWKJ7EABO5DTE6A5O4FCN4N26CETIDSG2TTKA4NLQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.706833209 (507h28m16.577916254s) [noise=true , address=45.91.116.7:4012     , piece_id=45LZ62SECG2WJYF2CHPKCRFXPCENYSANOJNL362IOIWOJMSGII6Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:16.822845286 (507h28m16.693928351s) [noise=true , address=45.91.116.13:4153    , piece_id=UVIQHZJPLLSQ4MJUMOZYYG77A4ZV2D4XDWTOVFBKUXJYJDLDLLAQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194874415 (507h28m17.06595745s) [noise=true , address=77.90.31.19:4844     , piece_id=ZOBMSSPERFTGXVVOD3CMREPUEG7OKQ23ZZFSC5KAV4MWPG5TY6NQ, seg_pos={0 0}, upload=445310]: Done uploading piece. err: <nil>
Dec 26 18:57:17.194899042 (507h28m17.065982077s) [seg_pos={0 0}, upload=445310]: Segment reached optimal threshold of 13 pieces.
Dec 26 18:57:17.195194714 (507h28m17.066277749s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57402->145.79.227.41:4085: use of closed network connection
Dec 26 18:57:17.195203200 (507h28m17.066286245s) [noise=true , address=145.79.227.41:4085   , piece_id=M6JE3XCWWJQNDW7H7ZLF2C36WZ6I4IYOLZPVFFVHCR5EUU3OX6PA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195257284 (507h28m17.066340319s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51280->45.91.116.4:4939: use of closed network connection
Dec 26 18:57:17.195271601 (507h28m17.066354636s) [noise=true , address=45.91.116.4:4939     , piece_id=CDT2I4DOLWWYGJOP27IJQC4WMRVWCHI2643YJELVLBV3TJRJV46Q, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195368315 (507h28m17.06645136s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk): protocol: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38342->145.79.227.19:4077: use of closed network connection
Dec 26 18:57:17.195374056 (507h28m17.066457091s) [noise=true , address=145.79.227.19:4077   , piece_id=7POI5LNWTJKQIDWMAKOIKBXJJLO5TYCG2YZRYYZOSWZCDGZ4YSAA, seg_pos={0 0}, upload=445310]: Long tail cancellation due to optimal threshold
Dec 26 18:57:17.195389205 (507h28m17.06647224s) [seg_pos={0 0}, upload=445310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 26 18:57:17.195465821 (507h28m17.066548866s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 26 18:57:17.195471041 (507h28m17.066554076s) [upload=445310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 26 18:57:17.496284783 (507h28m17.367367839s) [upload=445310]: Done uploading segments...
Dec 26 18:57:17.496304901 (507h28m17.367387936s) [upload=445310]: Upload finished. err: <nil>
Duration: 30.002792422s core uplink us-select-1