Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [....X......X...X....XX.............X..X.......X.............] duration: 5.785131724s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:16:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.785206555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....X......X...X....XX.............X..X.......X............] duration: 5.888642409s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.888679179s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.....X......X...X....XX.............X..X.......X...........] duration: 5.652969131s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:14:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.652996903s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X.....X......X...X....XX.............X..X.......X..........] duration: 5.68036099s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:14:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.68038813s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.....X......X...X....XX.............X..X.......X.........] duration: 5.833046269s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:13:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.833069514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.....X......X...X....XX.............X..X.......X........] duration: 5.857902294s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:13:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.857937311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.....X......X...X....XX.............X..X.......X.......] duration: 5.74878134s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:12:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.7488181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.....X......X...X....XX.............X..X.......X......] duration: 5.795185065s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:11:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.795211005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.....X......X...X....XX.............X..X.......X.....] duration: 5.696250994s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:10:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.696294818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X.....X......X...X....XX.............X..X.......X....] duration: 5.794849328s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:09:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.79487159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X.....X......X...X....XX.............X..X.......X...] duration: 5.826249193s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:09:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.826283197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X.....X......X...X....XX.............X..X.......X..] duration: 5.764025264s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:08:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.764052196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..........X.....X......X...X....XX.............X..X.......X.] duration: 5.869722434s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:08:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 5.869753042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...........X.....X......X...X....XX.............X..X.......X] duration: 30.002131303s trace: https://jaeger.dev.storj.io/trace/7eae229e1eab6fd7 FAIL (2025-12-26 01:07:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 01:07:09.581992421 (489h38m9.453075466s) [upload=429752]: Starting upload Dec 26 01:07:09.582297222 (489h38m9.453380257s) [upload=429752]: Done starting upload Dec 26 01:07:09.582435504 (489h38m9.453518539s) [upload=429752]: Uploading segments... Dec 26 01:07:09.582536908 (489h38m9.453619953s) [upload=429752]: Got next segment. Inline: false Dec 26 01:07:09.582570331 (489h38m9.453653376s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 01:07:09.582576322 (489h38m9.453659367s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:07:09.848649126 (489h38m9.719732171s) [seg_pos={0 0}, upload=429752]: Begin upload segment... Dec 26 01:07:09.848692538 (489h38m9.719775573s) [seg_pos={0 0}, upload=429752]: Done begin upload segment. Dec 26 01:07:09.848717245 (489h38m9.71980028s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848732014 (489h38m9.719815049s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.848842284 (489h38m9.719925329s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849366632 (489h38m9.720449677s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.849853870 (489h38m9.720936905s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850237670 (489h38m9.721320715s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850505019 (489h38m9.721588064s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.850771457 (489h38m9.721854502s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851130460 (489h38m9.722213495s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.851457182 (489h38m9.722540217s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.853188850 (489h38m9.724271895s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.854934643 (489h38m9.726017678s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.855794881 (489h38m9.726877926s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.856845962 (489h38m9.727928997s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.857955785 (489h38m9.72903882s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:09.859165188 (489h38m9.730248223s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Uploading piece... Dec 26 01:07:11.613699740 (489h38m11.484782786s) [upload=429752]: Next returned nil segment Dec 26 01:07:11.613718677 (489h38m11.484801722s) [upload=429752]: Waiting for error group managing segments... Dec 26 01:07:12.005403325 (489h38m11.876486381s) [noise=true , address=145.79.227.25:4617 , piece_id=NKZTDGMP5WXGJX73VLPY6XMYGBUHRH3EISVU6LSDFBVLFZ7VPBDA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.013856760 (489h38m11.884939795s) [noise=true , address=145.79.227.49:4337 , piece_id=7RG2LCLDCAVASB5ZJLKOJQLC2QIHPLYW7D2ZBJ32EURCTY4AN2QA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.014314662 (489h38m11.885397697s) [noise=true , address=145.79.227.50:4373 , piece_id=FK7UPVNACOQ7HCPWBDRXATFKVDFC2Z4D3E6676VCY6AHV5T3MWIQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018482346 (489h38m11.889565392s) [noise=true , address=145.79.227.35:4482 , piece_id=556A4NFLN2NV75D7CXDN3Q2ACW474MPWTTB5XETWEZEEG5QHZSLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018540347 (489h38m11.889623382s) [noise=true , address=145.79.227.29:4941 , piece_id=AJZEXE4WEAKEAZM5JGALRVHNKDT3ZM5T7RR2K75CKKZ363BFT45Q, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.018672760 (489h38m11.889755795s) [noise=true , address=145.79.227.14:4837 , piece_id=FT6F6R4GCZE7HBRYLJL5GIG2YBYJYU7IDNLYVA5ROG3IJJUMS53A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.022131134 (489h38m11.893214179s) [noise=true , address=145.79.227.52:4446 , piece_id=VZENAHZKSENSUIE6Z2BZHIZZNDFXO7B3I3TILPREOKFSK5REJIVQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.032104642 (489h38m11.903187677s) [noise=true , address=145.79.227.36:4517 , piece_id=UOV5WHYMCJSWQKX65XAMYHB2JX5AEOJUK5IU2BKJBSVNQQ6VLMUQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.035434562 (489h38m11.906517607s) [noise=true , address=145.79.227.38:4977 , piece_id=Y7Y7IYYDK2C7LARWF4D2J5IRB36ODAW6JVNOFS4K5I46NVBEM7CQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.041658081 (489h38m11.912741116s) [noise=true , address=145.79.227.42:4121 , piece_id=FBPSP74VAGRAJKG73R3BMGB6DKVII5MHNGG6BG27QGOKLLBUO2NA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.045270599 (489h38m11.916353634s) [noise=true , address=145.79.227.5:4223 , piece_id=6EVIQ7ONFO22RNOHDWF4L37GKYDQAZRTZ5VR6BVU6PGNXVVMRKLA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.050085527 (489h38m11.921168572s) [noise=true , address=45.91.116.10:4083 , piece_id=3SSQU7OGEMIL6I6B5B5TUY6RH4NRVLWNNAZ7O76PZIQJMLL54JIA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129262594 (489h38m12.000345639s) [noise=true , address=145.79.227.2:4114 , piece_id=BDY6MIUORPVJZ6VA6NEUNSJKGP7GTUTCNE2LVOBHTK6YUAA72NFA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: <nil> Dec 26 01:07:12.129285167 (489h38m12.000368202s) [seg_pos={0 0}, upload=429752]: Segment reached optimal threshold of 13 pieces. Dec 26 01:07:12.129629442 (489h38m12.000712487s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.129659509 (489h38m12.000742554s) [noise=true , address=145.79.227.41:4086 , piece_id=UX54METQMTEAS3DGQXTDCPIGIRISEYQ4LKRJWM65EYTMVNF64RKQ, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130184589 (489h38m12.001267634s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 01:07:12.130192906 (489h38m12.001275951s) [noise=true , address=45.91.116.9:4264 , piece_id=VKTUFSYMMKYOPJCBFO6IBZBLLYGDAW5YCG6QPOAO4NKPLSWDBI3A, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130401994 (489h38m12.001485029s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40688->77.90.31.19:4842: use of closed network connection Dec 26 01:07:12.130410941 (489h38m12.001493986s) [noise=true , address=77.90.31.19:4842 , piece_id=BJXAAXTX4FHYLI4KCF6F2FIRPMB4J5GN73YMTY3SYBKP6J6YHOQA, seg_pos={0 0}, upload=429752]: Long tail cancellation due to optimal threshold Dec 26 01:07:12.130428724 (489h38m12.001511769s) [seg_pos={0 0}, upload=429752]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:07:12.130504999 (489h38m12.001588044s) [upload=429752]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:07:12.130511121 (489h38m12.001594156s) [upload=429752]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:07:12.441918908 (489h38m12.313001943s) [upload=429752]: Done uploading segments... Dec 26 01:07:12.441940229 (489h38m12.313023264s) [upload=429752]: Upload finished. err: <nil> Duration: 30.002154537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X.....X......X...X....XX.............X..X.......] duration: 5.829024232s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:06:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.829053528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X.....X......X...X....XX.............X..X......] duration: 5.780636691s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:05:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.78066259s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.....X......X...X....XX.............X..X.....] duration: 5.79331603s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:05:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.793346238s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X.....X......X...X....XX.............X..X....] duration: 5.748550778s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:04:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.748578651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.....X......X...X....XX.............X..X...] duration: 5.87830374s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:03:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.878332836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.....X......X...X....XX.............X..X..] duration: 5.768890541s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:02:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.768922982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.....X......X...X....XX.............X..X.] duration: 5.96318315s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:01:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 5.963216033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.....X......X...X....XX.............X..X] duration: 30.002171696s trace: https://jaeger.dev.storj.io/trace/6d17a550e25f229b FAIL (2025-12-26 01:01:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 489h31m47.055649005s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 01:00:47.446570426 (489h31m47.317653471s) [seg_pos={0 0}, upload=429665]: Begin upload segment... Dec 26 01:00:47.446628197 (489h31m47.317711242s) [seg_pos={0 0}, upload=429665]: Done begin upload segment. Dec 26 01:00:47.446661190 (489h31m47.317744235s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447496721 (489h31m47.318579766s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.447927140 (489h31m47.319010185s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448314107 (489h31m47.319397142s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.448908659 (489h31m47.319991694s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449297759 (489h31m47.320380804s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.449713571 (489h31m47.320796616s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450350063 (489h31m47.321433108s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.450738342 (489h31m47.321821377s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451138964 (489h31m47.322222s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451521653 (489h31m47.322604688s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.451887991 (489h31m47.322971036s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452279657 (489h31m47.323362702s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.452679628 (489h31m47.323762663s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453049062 (489h31m47.324132097s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:47.453512133 (489h31m47.324595168s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Uploading piece... Dec 26 01:00:49.232816225 (489h31m49.10389927s) [upload=429665]: Next returned nil segment Dec 26 01:00:49.233056081 (489h31m49.104139126s) [upload=429665]: Waiting for error group managing segments... Dec 26 01:00:49.577519925 (489h31m49.44860297s) [noise=true , address=77.90.31.11:4556 , piece_id=NKZUF4XWLV65CACN7Y7NORPVMZGZOMMYGJ2LGSLGLU7ESHF6K7BQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.588910111 (489h31m49.459993146s) [noise=true , address=145.79.227.6:4259 , piece_id=TYL5YMP5IQY3QIXYIP2QHP67JHCINGZ6LOYLAAAOBZ4PIMMXZO7Q, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.600959001 (489h31m49.472042046s) [noise=true , address=77.90.31.14:4661 , piece_id=YYDTDAWYVSYLFXBDLQHEGRRKFSJGP4FPNWLWAXRW24ABBNY7ANQA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.605508303 (489h31m49.476591338s) [noise=true , address=145.79.227.15:4876 , piece_id=XKZPXZIRTGASMXR27SPA53FZDIULS3LBJQO4H7YZEWZWOG45KUWQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.608688287 (489h31m49.479771333s) [noise=true , address=145.79.227.37:4566 , piece_id=HTTVKOPUHIKKPFU5R344OAOYSR6RQBF2SE6DSXK4YD7KV6TRUESA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.623933313 (489h31m49.495016358s) [noise=true , address=145.79.227.52:4447 , piece_id=RXDUX346TWQFF7XMT2XL5TXWT7VEP4Y4IOMR2UMAF6JYPEMEXDEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636405719 (489h31m49.507488764s) [noise=true , address=145.79.227.51:4410 , piece_id=URVDV4I2G3OM3SDKEL4GTU62XXD2MR225W2H6RI73PRSCIL5E6PQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.636430455 (489h31m49.5075135s) [noise=true , address=145.79.227.41:4086 , piece_id=VAE3C6GSFIGG7VKQZDSCCDUSYKWTVFW7EMNMJ5SOYXPQBNKYKYEQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.641725898 (489h31m49.512808933s) [noise=true , address=45.91.116.13:4154 , piece_id=OMLWLPF2ZQ6GUC5AIOZZYD75HH6PNPDKG7AHNZ7UMY27YKKCVM4A, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.664878589 (489h31m49.535961634s) [noise=true , address=45.91.116.4:4937 , piece_id=YDCBH74UETREHNGMSVO52EPK6VP2ANOBXUWGYS6RMITZ6C44V2SA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.672213914 (489h31m49.543296949s) [noise=true , address=77.90.31.21:4878 , piece_id=EE2WHQNJM3B4GTHHPYM3S6YLJYJ37A4YLO3TRCRDGK3O67NSORLA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.673280796 (489h31m49.544363871s) [noise=true , address=145.79.227.45:4231 , piece_id=NBBR55ZKWSAQA2YDIWHWZ52ERIDLCCJRLI7G6TYVLSSJBWDHERRQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885004802 (489h31m49.756087837s) [noise=true , address=145.79.227.14:4838 , piece_id=BCAVIXKXONSUMO6R7QCCV2VFWURBUEFVND57AQIYVOTRICWS43DA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: <nil> Dec 26 01:00:49.885028227 (489h31m49.756111272s) [seg_pos={0 0}, upload=429665]: Segment reached optimal threshold of 13 pieces. Dec 26 01:00:49.885195125 (489h31m49.75627817s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3): protocol: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43638->145.79.227.36:4519: use of closed network connection Dec 26 01:00:49.885203140 (489h31m49.756286175s) [noise=true , address=145.79.227.36:4519 , piece_id=QTO3CK6KEBI3ZKRJC26ZT5UQRONO7MQ7BA7UJPAZVYRTN3OKXVFA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885417208 (489h31m49.756500253s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45500->145.79.227.16:4910: use of closed network connection Dec 26 01:00:49.885424071 (489h31m49.756507106s) [noise=true , address=145.79.227.16:4910 , piece_id=4KG2NJNBE5AERO6SX2E4SZ4VQFSZNWRWVRIMO5YETGWYBV74UJNA, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885542887 (489h31m49.756625922s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi): protocol: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45928->145.79.227.42:4122: use of closed network connection Dec 26 01:00:49.885549019 (489h31m49.756632054s) [noise=true , address=145.79.227.42:4122 , piece_id=EEBXJ3HEZGU7R7AMBGJDWVAX5LNRIXIGM7JBB3CQRUDZG4XWKRZQ, seg_pos={0 0}, upload=429665]: Long tail cancellation due to optimal threshold Dec 26 01:00:49.885565891 (489h31m49.756648936s) [seg_pos={0 0}, upload=429665]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 01:00:49.885631956 (489h31m49.756714991s) [upload=429665]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 01:00:49.885636345 (489h31m49.75671938s) [upload=429665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 01:00:50.183635313 (489h31m50.054718348s) [upload=429665]: Done uploading segments... Dec 26 01:00:50.183661603 (489h31m50.054744648s) [upload=429665]: Upload finished. err: <nil> Duration:* 30.00219514s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....................X.....X......X...X....XX.............X..] duration: 5.711622252s trace: https://jaeger.dev.storj.io/trace/1fb5c520316418e8 FAIL (2025-12-26 00:59:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7:42.121422787 (489h28m41.992505822s) [upload=429622]: Starting upload Dec 26 00:57:42.121492560 (489h28m41.992575605s) [upload=429622]: Done starting upload Dec 26 00:57:42.121576970 (489h28m41.992660005s) [upload=429622]: Uploading segments... Dec 26 00:57:42.121830514 (489h28m41.992913559s) [upload=429622]: Got next segment. Inline: false Dec 26 00:57:42.121868396 (489h28m41.992951431s) [upload=429622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:57:42.121873114 (489h28m41.992956149s) [upload=429622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:57:42.382070419 (489h28m42.253153455s) [seg_pos={0 0}, upload=429622]: Begin upload segment... Dec 26 00:57:42.382116598 (489h28m42.253199633s) [seg_pos={0 0}, upload=429622]: Done begin upload segment. Dec 26 00:57:42.382141706 (489h28m42.253224741s) [noise=true , address=145.79.227.48:4303 , piece_id=WGGDV32WR5PKAO6XZ2PT2YGPJQ4H4KCUNPBGZKCZ5RMLWFO6SKJQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382216669 (489h28m42.253299714s) [noise=true , address=145.79.227.13:4802 , piece_id=SS5XGGM5LHHH3RW7CCNVZ6DSPMDY4GLOKHLLF3VAXOGG4TZXVUSQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382669912 (489h28m42.253752947s) [noise=true , address=145.79.227.42:4123 , piece_id=SGPJTFQN4GNZATMTY7HDQ3GQWDGMWWMJXRRMZGBHAZA43NT62AHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382996423 (489h28m42.254079468s) [noise=true , address=45.91.116.8:4047 , piece_id=FICFDJ6NBWXKA3OYEMSOEHLA6P4GLA3VA2S2332IQ2RMKIIHLHKA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383220339 (489h28m42.254303384s) [noise=true , address=145.79.227.11:4730 , piece_id=BHXTYV3VKE675UFINWLEE3CF3TGZWUATYVYRINYPHL3H5KZHVJHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383565456 (489h28m42.254648491s) [noise=true , address=77.90.31.16:4736 , piece_id=FNVR4OQ3WEQ5K5VTY2HIX3RMHQPMK23H45PETFGHXA7G3VMOPW5A, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383906506 (489h28m42.254989541s) [noise=true , address=145.79.227.49:4337 , piece_id=P56Z4OB7P4SLJ3ACEDWKCD4XSZXCQ7RX4R3N52XQDP25RJ3ZJR4A, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384291830 (489h28m42.255374865s) [noise=true , address=145.79.227.50:4373 , piece_id=SL7QTHCR64ZJ3HSMD5ISJNTH437RX55I26J2GKUEHU4E2TBCGLQQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384364829 (489h28m42.255447864s) [noise=true , address=145.79.227.32:4834 , piece_id=NNSIRRBFYIITOMMGOTI7WAI46ERZGN655Y6MOYYPKG6ZZNHP4U4Q, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384932720 (489h28m42.256015755s) [noise=true , address=145.79.227.37:4567 , piece_id=2L4VJ4OM7VMEL7W2NWMXJFH6Q2H6P35Q4UWLNR4WEYOOZEFEEOLA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384989238 (489h28m42.256072273s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.386030539 (489h28m42.257113585s) [noise=true , address=145.79.227.38:4978 , piece_id=OCOKGAOLKWALKY2B2VJE3XPD64RGKWMNJ66CCLHU4SBCGHKWNLFQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.386467692 (489h28m42.257642582s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.388026049 (489h28m42.259109094s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.388426933 (489h28m42.259509968s) [noise=true , address=77.90.31.15:4700 , piece_id=RQP5N3IBF33NPGWXEPSJVULDDFXY6GIGZBFS3MXXKR3K7KIDUSTA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.391291516 (489h28m42.262374561s) [noise=true , address=145.79.227.17:4971 , piece_id=2YQUEP3V72TVDXOZY5J4XQGIPPZSL4KI7EMAS7CU2ZNEUTBXEMHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:44.205252221 (489h28m44.076335256s) [upload=429622]: Next returned nil segment Dec 26 00:57:44.205266449 (489h28m44.076349484s) [upload=429622]: Waiting for error group managing segments... Dec 26 00:57:44.518855759 (489h28m44.389938814s) [noise=true , address=77.90.31.16:4736 , piece_id=FNVR4OQ3WEQ5K5VTY2HIX3RMHQPMK23H45PETFGHXA7G3VMOPW5A, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.538064441 (489h28m44.409147476s) [noise=true , address=145.79.227.48:4303 , piece_id=WGGDV32WR5PKAO6XZ2PT2YGPJQ4H4KCUNPBGZKCZ5RMLWFO6SKJQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.538168679 (489h28m44.409251724s) [noise=true , address=145.79.227.37:4567 , piece_id=2L4VJ4OM7VMEL7W2NWMXJFH6Q2H6P35Q4UWLNR4WEYOOZEFEEOLA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.540694588 (489h28m44.411777623s) [noise=true , address=77.90.31.15:4700 , piece_id=RQP5N3IBF33NPGWXEPSJVULDDFXY6GIGZBFS3MXXKR3K7KIDUSTA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.552841786 (489h28m44.423924821s) [noise=true , address=145.79.227.11:4730 , piece_id=BHXTYV3VKE675UFINWLEE3CF3TGZWUATYVYRINYPHL3H5KZHVJHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.553081432 (489h28m44.424164467s) [noise=true , address=145.79.227.49:4337 , piece_id=P56Z4OB7P4SLJ3ACEDWKCD4XSZXCQ7RX4R3N52XQDP25RJ3ZJR4A, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.559703940 (489h28m44.430786985s) [noise=true , address=145.79.227.42:4123 , piece_id=SGPJTFQN4GNZATMTY7HDQ3GQWDGMWWMJXRRMZGBHAZA43NT62AHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.561568570 (489h28m44.432651605s) [noise=true , address=45.91.116.8:4047 , piece_id=FICFDJ6NBWXKA3OYEMSOEHLA6P4GLA3VA2S2332IQ2RMKIIHLHKA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.565582613 (489h28m44.436665659s) [noise=true , address=145.79.227.50:4373 , piece_id=SL7QTHCR64ZJ3HSMD5ISJNTH437RX55I26J2GKUEHU4E2TBCGLQQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.569321552 (489h28m44.440404587s) [noise=true , address=145.79.227.17:4971 , piece_id=2YQUEP3V72TVDXOZY5J4XQGIPPZSL4KI7EMAS7CU2ZNEUTBXEMHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.570121274 (489h28m44.44120431s) [noise=true , address=145.79.227.38:4978 , piece_id=OCOKGAOLKWALKY2B2VJE3XPD64RGKWMNJ66CCLHU4SBCGHKWNLFQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.573679409 (489h28m44.444762444s) [noise=true , address=145.79.227.32:4834 , piece_id=NNSIRRBFYIITOMMGOTI7WAI46ERZGN655Y6MOYYPKG6ZZNHP4U4Q, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.590399193 (489h28m44.461482228s) [noise=true , address=145.79.227.13:4802 , piece_id=SS5XGGM5LHHH3RW7CCNVZ6DSPMDY4GLOKHLLF3VAXOGG4TZXVUSQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.590418689 (489h28m44.461501734s) [seg_pos={0 0}, upload=429622]: Segment reached optimal threshold of 13 pieces. Dec 26 00:57:44.590586238 (489h28m44.461669273s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 00:57:44.590592490 (489h28m44.461675545s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590834842 (489h28m44.461917887s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection Dec 26 00:57:44.590848228 (489h28m44.461931273s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590930835 (489h28m44.46201387s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 00:57:44.590939170 (489h28m44.462022215s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590957726 (489h28m44.462040761s) [seg_pos={0 0}, upload=429622]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:57:44.591063067 (489h28m44.462146102s) [upload=429622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:57:44.591070691 (489h28m44.462153746s) [upload=429622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:57:44.901638287 (489h28m44.772721332s) [upload=429622]: Done uploading segments... Dec 26 00:57:44.901655871 (489h28m44.772738916s) [upload=429622]: Upload finished. err: <nil> Duration: 5.7116466s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....................X.....X......X...X....XX.............X.] duration: 5.7711831s trace: https://jaeger.dev.storj.io/trace/1fb5c520316418e8 FAIL (2025-12-26 00:58:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7:42.121422787 (489h28m41.992505822s) [upload=429622]: Starting upload Dec 26 00:57:42.121492560 (489h28m41.992575605s) [upload=429622]: Done starting upload Dec 26 00:57:42.121576970 (489h28m41.992660005s) [upload=429622]: Uploading segments... Dec 26 00:57:42.121830514 (489h28m41.992913559s) [upload=429622]: Got next segment. Inline: false Dec 26 00:57:42.121868396 (489h28m41.992951431s) [upload=429622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:57:42.121873114 (489h28m41.992956149s) [upload=429622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:57:42.382070419 (489h28m42.253153455s) [seg_pos={0 0}, upload=429622]: Begin upload segment... Dec 26 00:57:42.382116598 (489h28m42.253199633s) [seg_pos={0 0}, upload=429622]: Done begin upload segment. Dec 26 00:57:42.382141706 (489h28m42.253224741s) [noise=true , address=145.79.227.48:4303 , piece_id=WGGDV32WR5PKAO6XZ2PT2YGPJQ4H4KCUNPBGZKCZ5RMLWFO6SKJQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382216669 (489h28m42.253299714s) [noise=true , address=145.79.227.13:4802 , piece_id=SS5XGGM5LHHH3RW7CCNVZ6DSPMDY4GLOKHLLF3VAXOGG4TZXVUSQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382669912 (489h28m42.253752947s) [noise=true , address=145.79.227.42:4123 , piece_id=SGPJTFQN4GNZATMTY7HDQ3GQWDGMWWMJXRRMZGBHAZA43NT62AHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382996423 (489h28m42.254079468s) [noise=true , address=45.91.116.8:4047 , piece_id=FICFDJ6NBWXKA3OYEMSOEHLA6P4GLA3VA2S2332IQ2RMKIIHLHKA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383220339 (489h28m42.254303384s) [noise=true , address=145.79.227.11:4730 , piece_id=BHXTYV3VKE675UFINWLEE3CF3TGZWUATYVYRINYPHL3H5KZHVJHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383565456 (489h28m42.254648491s) [noise=true , address=77.90.31.16:4736 , piece_id=FNVR4OQ3WEQ5K5VTY2HIX3RMHQPMK23H45PETFGHXA7G3VMOPW5A, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383906506 (489h28m42.254989541s) [noise=true , address=145.79.227.49:4337 , piece_id=P56Z4OB7P4SLJ3ACEDWKCD4XSZXCQ7RX4R3N52XQDP25RJ3ZJR4A, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384291830 (489h28m42.255374865s) [noise=true , address=145.79.227.50:4373 , piece_id=SL7QTHCR64ZJ3HSMD5ISJNTH437RX55I26J2GKUEHU4E2TBCGLQQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384364829 (489h28m42.255447864s) [noise=true , address=145.79.227.32:4834 , piece_id=NNSIRRBFYIITOMMGOTI7WAI46ERZGN655Y6MOYYPKG6ZZNHP4U4Q, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384932720 (489h28m42.256015755s) [noise=true , address=145.79.227.37:4567 , piece_id=2L4VJ4OM7VMEL7W2NWMXJFH6Q2H6P35Q4UWLNR4WEYOOZEFEEOLA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384989238 (489h28m42.256072273s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.386030539 (489h28m42.257113585s) [noise=true , address=145.79.227.38:4978 , piece_id=OCOKGAOLKWALKY2B2VJE3XPD64RGKWMNJ66CCLHU4SBCGHKWNLFQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.386467692 (489h28m42.257642582s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.388026049 (489h28m42.259109094s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.388426933 (489h28m42.259509968s) [noise=true , address=77.90.31.15:4700 , piece_id=RQP5N3IBF33NPGWXEPSJVULDDFXY6GIGZBFS3MXXKR3K7KIDUSTA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.391291516 (489h28m42.262374561s) [noise=true , address=145.79.227.17:4971 , piece_id=2YQUEP3V72TVDXOZY5J4XQGIPPZSL4KI7EMAS7CU2ZNEUTBXEMHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:44.205252221 (489h28m44.076335256s) [upload=429622]: Next returned nil segment Dec 26 00:57:44.205266449 (489h28m44.076349484s) [upload=429622]: Waiting for error group managing segments... Dec 26 00:57:44.518855759 (489h28m44.389938814s) [noise=true , address=77.90.31.16:4736 , piece_id=FNVR4OQ3WEQ5K5VTY2HIX3RMHQPMK23H45PETFGHXA7G3VMOPW5A, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.538064441 (489h28m44.409147476s) [noise=true , address=145.79.227.48:4303 , piece_id=WGGDV32WR5PKAO6XZ2PT2YGPJQ4H4KCUNPBGZKCZ5RMLWFO6SKJQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.538168679 (489h28m44.409251724s) [noise=true , address=145.79.227.37:4567 , piece_id=2L4VJ4OM7VMEL7W2NWMXJFH6Q2H6P35Q4UWLNR4WEYOOZEFEEOLA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.540694588 (489h28m44.411777623s) [noise=true , address=77.90.31.15:4700 , piece_id=RQP5N3IBF33NPGWXEPSJVULDDFXY6GIGZBFS3MXXKR3K7KIDUSTA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.552841786 (489h28m44.423924821s) [noise=true , address=145.79.227.11:4730 , piece_id=BHXTYV3VKE675UFINWLEE3CF3TGZWUATYVYRINYPHL3H5KZHVJHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.553081432 (489h28m44.424164467s) [noise=true , address=145.79.227.49:4337 , piece_id=P56Z4OB7P4SLJ3ACEDWKCD4XSZXCQ7RX4R3N52XQDP25RJ3ZJR4A, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.559703940 (489h28m44.430786985s) [noise=true , address=145.79.227.42:4123 , piece_id=SGPJTFQN4GNZATMTY7HDQ3GQWDGMWWMJXRRMZGBHAZA43NT62AHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.561568570 (489h28m44.432651605s) [noise=true , address=45.91.116.8:4047 , piece_id=FICFDJ6NBWXKA3OYEMSOEHLA6P4GLA3VA2S2332IQ2RMKIIHLHKA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.565582613 (489h28m44.436665659s) [noise=true , address=145.79.227.50:4373 , piece_id=SL7QTHCR64ZJ3HSMD5ISJNTH437RX55I26J2GKUEHU4E2TBCGLQQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.569321552 (489h28m44.440404587s) [noise=true , address=145.79.227.17:4971 , piece_id=2YQUEP3V72TVDXOZY5J4XQGIPPZSL4KI7EMAS7CU2ZNEUTBXEMHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.570121274 (489h28m44.44120431s) [noise=true , address=145.79.227.38:4978 , piece_id=OCOKGAOLKWALKY2B2VJE3XPD64RGKWMNJ66CCLHU4SBCGHKWNLFQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.573679409 (489h28m44.444762444s) [noise=true , address=145.79.227.32:4834 , piece_id=NNSIRRBFYIITOMMGOTI7WAI46ERZGN655Y6MOYYPKG6ZZNHP4U4Q, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.590399193 (489h28m44.461482228s) [noise=true , address=145.79.227.13:4802 , piece_id=SS5XGGM5LHHH3RW7CCNVZ6DSPMDY4GLOKHLLF3VAXOGG4TZXVUSQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.590418689 (489h28m44.461501734s) [seg_pos={0 0}, upload=429622]: Segment reached optimal threshold of 13 pieces. Dec 26 00:57:44.590586238 (489h28m44.461669273s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 00:57:44.590592490 (489h28m44.461675545s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590834842 (489h28m44.461917887s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection Dec 26 00:57:44.590848228 (489h28m44.461931273s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590930835 (489h28m44.46201387s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 00:57:44.590939170 (489h28m44.462022215s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590957726 (489h28m44.462040761s) [seg_pos={0 0}, upload=429622]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:57:44.591063067 (489h28m44.462146102s) [upload=429622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:57:44.591070691 (489h28m44.462153746s) [upload=429622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:57:44.901638287 (489h28m44.772721332s) [upload=429622]: Done uploading segments... Dec 26 00:57:44.901655871 (489h28m44.772738916s) [upload=429622]: Upload finished. err: <nil> Duration: 5.771204251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......................X.....X......X...X....XX.............X] duration: 30.070102488s trace: https://jaeger.dev.storj.io/trace/1fb5c520316418e8 FAIL (2025-12-26 00:58:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7:42.121422787 (489h28m41.992505822s) [upload=429622]: Starting upload Dec 26 00:57:42.121492560 (489h28m41.992575605s) [upload=429622]: Done starting upload Dec 26 00:57:42.121576970 (489h28m41.992660005s) [upload=429622]: Uploading segments... Dec 26 00:57:42.121830514 (489h28m41.992913559s) [upload=429622]: Got next segment. Inline: false Dec 26 00:57:42.121868396 (489h28m41.992951431s) [upload=429622]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:57:42.121873114 (489h28m41.992956149s) [upload=429622]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:57:42.382070419 (489h28m42.253153455s) [seg_pos={0 0}, upload=429622]: Begin upload segment... Dec 26 00:57:42.382116598 (489h28m42.253199633s) [seg_pos={0 0}, upload=429622]: Done begin upload segment. Dec 26 00:57:42.382141706 (489h28m42.253224741s) [noise=true , address=145.79.227.48:4303 , piece_id=WGGDV32WR5PKAO6XZ2PT2YGPJQ4H4KCUNPBGZKCZ5RMLWFO6SKJQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382216669 (489h28m42.253299714s) [noise=true , address=145.79.227.13:4802 , piece_id=SS5XGGM5LHHH3RW7CCNVZ6DSPMDY4GLOKHLLF3VAXOGG4TZXVUSQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382669912 (489h28m42.253752947s) [noise=true , address=145.79.227.42:4123 , piece_id=SGPJTFQN4GNZATMTY7HDQ3GQWDGMWWMJXRRMZGBHAZA43NT62AHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.382996423 (489h28m42.254079468s) [noise=true , address=45.91.116.8:4047 , piece_id=FICFDJ6NBWXKA3OYEMSOEHLA6P4GLA3VA2S2332IQ2RMKIIHLHKA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383220339 (489h28m42.254303384s) [noise=true , address=145.79.227.11:4730 , piece_id=BHXTYV3VKE675UFINWLEE3CF3TGZWUATYVYRINYPHL3H5KZHVJHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383565456 (489h28m42.254648491s) [noise=true , address=77.90.31.16:4736 , piece_id=FNVR4OQ3WEQ5K5VTY2HIX3RMHQPMK23H45PETFGHXA7G3VMOPW5A, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.383906506 (489h28m42.254989541s) [noise=true , address=145.79.227.49:4337 , piece_id=P56Z4OB7P4SLJ3ACEDWKCD4XSZXCQ7RX4R3N52XQDP25RJ3ZJR4A, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384291830 (489h28m42.255374865s) [noise=true , address=145.79.227.50:4373 , piece_id=SL7QTHCR64ZJ3HSMD5ISJNTH437RX55I26J2GKUEHU4E2TBCGLQQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384364829 (489h28m42.255447864s) [noise=true , address=145.79.227.32:4834 , piece_id=NNSIRRBFYIITOMMGOTI7WAI46ERZGN655Y6MOYYPKG6ZZNHP4U4Q, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384932720 (489h28m42.256015755s) [noise=true , address=145.79.227.37:4567 , piece_id=2L4VJ4OM7VMEL7W2NWMXJFH6Q2H6P35Q4UWLNR4WEYOOZEFEEOLA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.384989238 (489h28m42.256072273s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.386030539 (489h28m42.257113585s) [noise=true , address=145.79.227.38:4978 , piece_id=OCOKGAOLKWALKY2B2VJE3XPD64RGKWMNJ66CCLHU4SBCGHKWNLFQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.386467692 (489h28m42.257642582s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.388026049 (489h28m42.259109094s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.388426933 (489h28m42.259509968s) [noise=true , address=77.90.31.15:4700 , piece_id=RQP5N3IBF33NPGWXEPSJVULDDFXY6GIGZBFS3MXXKR3K7KIDUSTA, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:42.391291516 (489h28m42.262374561s) [noise=true , address=145.79.227.17:4971 , piece_id=2YQUEP3V72TVDXOZY5J4XQGIPPZSL4KI7EMAS7CU2ZNEUTBXEMHQ, seg_pos={0 0}, upload=429622]: Uploading piece... Dec 26 00:57:44.205252221 (489h28m44.076335256s) [upload=429622]: Next returned nil segment Dec 26 00:57:44.205266449 (489h28m44.076349484s) [upload=429622]: Waiting for error group managing segments... Dec 26 00:57:44.518855759 (489h28m44.389938814s) [noise=true , address=77.90.31.16:4736 , piece_id=FNVR4OQ3WEQ5K5VTY2HIX3RMHQPMK23H45PETFGHXA7G3VMOPW5A, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.538064441 (489h28m44.409147476s) [noise=true , address=145.79.227.48:4303 , piece_id=WGGDV32WR5PKAO6XZ2PT2YGPJQ4H4KCUNPBGZKCZ5RMLWFO6SKJQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.538168679 (489h28m44.409251724s) [noise=true , address=145.79.227.37:4567 , piece_id=2L4VJ4OM7VMEL7W2NWMXJFH6Q2H6P35Q4UWLNR4WEYOOZEFEEOLA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.540694588 (489h28m44.411777623s) [noise=true , address=77.90.31.15:4700 , piece_id=RQP5N3IBF33NPGWXEPSJVULDDFXY6GIGZBFS3MXXKR3K7KIDUSTA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.552841786 (489h28m44.423924821s) [noise=true , address=145.79.227.11:4730 , piece_id=BHXTYV3VKE675UFINWLEE3CF3TGZWUATYVYRINYPHL3H5KZHVJHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.553081432 (489h28m44.424164467s) [noise=true , address=145.79.227.49:4337 , piece_id=P56Z4OB7P4SLJ3ACEDWKCD4XSZXCQ7RX4R3N52XQDP25RJ3ZJR4A, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.559703940 (489h28m44.430786985s) [noise=true , address=145.79.227.42:4123 , piece_id=SGPJTFQN4GNZATMTY7HDQ3GQWDGMWWMJXRRMZGBHAZA43NT62AHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.561568570 (489h28m44.432651605s) [noise=true , address=45.91.116.8:4047 , piece_id=FICFDJ6NBWXKA3OYEMSOEHLA6P4GLA3VA2S2332IQ2RMKIIHLHKA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.565582613 (489h28m44.436665659s) [noise=true , address=145.79.227.50:4373 , piece_id=SL7QTHCR64ZJ3HSMD5ISJNTH437RX55I26J2GKUEHU4E2TBCGLQQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.569321552 (489h28m44.440404587s) [noise=true , address=145.79.227.17:4971 , piece_id=2YQUEP3V72TVDXOZY5J4XQGIPPZSL4KI7EMAS7CU2ZNEUTBXEMHQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.570121274 (489h28m44.44120431s) [noise=true , address=145.79.227.38:4978 , piece_id=OCOKGAOLKWALKY2B2VJE3XPD64RGKWMNJ66CCLHU4SBCGHKWNLFQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.573679409 (489h28m44.444762444s) [noise=true , address=145.79.227.32:4834 , piece_id=NNSIRRBFYIITOMMGOTI7WAI46ERZGN655Y6MOYYPKG6ZZNHP4U4Q, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.590399193 (489h28m44.461482228s) [noise=true , address=145.79.227.13:4802 , piece_id=SS5XGGM5LHHH3RW7CCNVZ6DSPMDY4GLOKHLLF3VAXOGG4TZXVUSQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: <nil> Dec 26 00:57:44.590418689 (489h28m44.461501734s) [seg_pos={0 0}, upload=429622]: Segment reached optimal threshold of 13 pieces. Dec 26 00:57:44.590586238 (489h28m44.461669273s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 00:57:44.590592490 (489h28m44.461675545s) [noise=true , address=45.91.116.10:4083 , piece_id=GZBRBLUPBG7J4H5FP5ABCR3GMQDOY2QENZS5NGA66YEDA6KUGWCQ, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590834842 (489h28m44.461917887s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55410->145.79.227.15:4876: use of closed network connection Dec 26 00:57:44.590848228 (489h28m44.461931273s) [noise=true , address=145.79.227.15:4876 , piece_id=2WTGN6QPQZ75GJTCHZOD2IJUX73ZZVBZULSVC65FUVJZTMFR23KA, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590930835 (489h28m44.46201387s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 26 00:57:44.590939170 (489h28m44.462022215s) [noise=true , address=45.91.116.11:4119 , piece_id=MEGN6GTHJ7NEEEW27RKY73UBOOYMDK4HOS3E7QKVV5XEIBQBURIA, seg_pos={0 0}, upload=429622]: Long tail cancellation due to optimal threshold Dec 26 00:57:44.590957726 (489h28m44.462040761s) [seg_pos={0 0}, upload=429622]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:57:44.591063067 (489h28m44.462146102s) [upload=429622]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:57:44.591070691 (489h28m44.462153746s) [upload=429622]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:57:44.901638287 (489h28m44.772721332s) [upload=429622]: Done uploading segments... Dec 26 00:57:44.901655871 (489h28m44.772738916s) [upload=429622]: Upload finished. err: <nil> Duration: 30.070125862s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X.....X......X...X....XX.............] duration: 5.910614622s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:56:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.91065592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X.....X......X...X....XX............] duration: 5.80475546s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:56:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.804783372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X.....X......X...X....XX...........] duration: 5.777240012s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:54:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.777267796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X.....X......X...X....XX..........] duration: 5.80643675s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.806466266s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X.....X......X...X....XX.........] duration: 5.906724022s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:53:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.906759279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X.....X......X...X....XX........] duration: 5.826456795s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:52:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.826481592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X.....X......X...X....XX.......] duration: 5.752648912s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.752677767s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................X.....X......X...X....XX......] duration: 5.650701253s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:51:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.650732503s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................X.....X......X...X....XX.....] duration: 17.995460327s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:50:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 17.995484773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................................X.....X......X...X....XX....] duration: 5.712440195s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:49:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.712465763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................................X.....X......X...X....XX...] duration: 5.810149516s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:49:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.810182178s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................................X.....X......X...X....XX..] duration: 5.662115823s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.662148014s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................................X.....X......X...X....XX.] duration: 5.769359138s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:48:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 5.769392671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................................X.....X......X...X....XX] duration: 30.00231289s trace: https://jaeger.dev.storj.io/trace/4c0e4b65582f6089 FAIL (2025-12-26 00:47:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 13.606109069 (489h18m13.477192104s) [upload=429469]: Done starting upload Dec 26 00:47:13.606241191 (489h18m13.477324236s) [upload=429469]: Uploading segments... Dec 26 00:47:13.606395204 (489h18m13.477478249s) [upload=429469]: Got next segment. Inline: false Dec 26 00:47:13.606446822 (489h18m13.477529857s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:47:13.606453776 (489h18m13.477536811s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:47:13.851262247 (489h18m13.722345282s) [seg_pos={0 0}, upload=429469]: Begin upload segment... Dec 26 00:47:13.851325487 (489h18m13.722408522s) [seg_pos={0 0}, upload=429469]: Done begin upload segment. Dec 26 00:47:13.851380142 (489h18m13.722463177s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.851462528 (489h18m13.722545563s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852024198 (489h18m13.723107233s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852319571 (489h18m13.723402616s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852615243 (489h18m13.723698288s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.852927418 (489h18m13.724010453s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853301450 (489h18m13.724384485s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853615388 (489h18m13.724698423s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.853995452 (489h18m13.725078487s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.854857112 (489h18m13.725940157s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.856134274 (489h18m13.727217309s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.857857074 (489h18m13.728940109s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.860118490 (489h18m13.731201525s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.862147674 (489h18m13.733230709s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.863508485 (489h18m13.73459152s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:13.865738000 (489h18m13.736821035s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Uploading piece... Dec 26 00:47:15.669759293 (489h18m15.540842348s) [upload=429469]: Next returned nil segment Dec 26 00:47:15.669771456 (489h18m15.540854491s) [upload=429469]: Waiting for error group managing segments... Dec 26 00:47:16.003099241 (489h18m15.874182276s) [noise=true , address=77.90.31.17:4770 , piece_id=GNP75VVHQGOALZ4AWLCMBX4TQOX2PS3RLAGLHTKSAH4ZJZFAUMSQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.012685484 (489h18m15.883768519s) [noise=true , address=145.79.227.42:4122 , piece_id=4WGXOEKTUVOV3JQWUWAZIX6A4XQ7EF46ELJNZ5BYD36OAX6A3VQA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.020051488 (489h18m15.891134523s) [noise=true , address=145.79.227.45:4229 , piece_id=BYSFT4AFCCTXVVZJWUOESWG3F4IV5JI6K56GWNZKHRCA5ELG7ZJQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.021956536 (489h18m15.893039581s) [noise=true , address=145.79.227.17:4972 , piece_id=QTY7ZCT5R5IXDG5ECHYGB2ZFYH3HVJLPXUVZLTP44QVDOBETYWCA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.024416099 (489h18m15.895499134s) [noise=true , address=45.91.116.13:4154 , piece_id=W5HH45ZVPQNOGZKCTDGWZEAVRX76GBFKDJ4EHPBN3VAODG5LGUZA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.027907787 (489h18m15.898990822s) [noise=true , address=45.91.116.6:4975 , piece_id=25WMETE255BBZPFP4ZZULLXCWWM2VSVUWHEI77O55QNUUDLTVE4Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.033835934 (489h18m15.904918969s) [noise=true , address=145.79.227.37:4565 , piece_id=CAWE5M372AZJZAEJFOXHH77NYFU3AP43JIWS7SJOSZUPJF5UOPOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.051490559 (489h18m15.922573594s) [noise=true , address=145.79.227.15:4873 , piece_id=VDZ3R56NXQ3WQ7CFHDGWS6Q66Y47BUVCCG5SDLRFX3YOR6SB34NA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.053126074 (489h18m15.924209109s) [noise=true , address=45.91.116.4:4937 , piece_id=WUX5DF5ROXV4PKKT6BSVFORNSLQRSRPHL6P4J7VOIFJRMJ5BGKHA, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.078822861 (489h18m15.949905906s) [noise=true , address=145.79.227.11:4731 , piece_id=CV3IPURMBOL2BMTW3HIV4SSFUQ3JTEGSCQ6GWQH5AU746H4FYVYQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.125923361 (489h18m15.997006406s) [noise=true , address=45.91.116.12:4228 , piece_id=J4QQUOOTL2ILYWXFDF5VFC56PKHLL7HQYU54CR7BSPPQXZIT7HFQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.316177560 (489h18m16.187260605s) [noise=true , address=145.79.227.39:4013 , piece_id=BR4OQY5W7VF55A2JYZ4BXDCVDKOCZNSHVJ6E4I2ICN2H7XFAYQUQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722522672 (489h18m16.593605707s) [noise=true , address=45.91.116.14:4192 , piece_id=EWVLPDLOR2CKD2LX37HQFO7CDCELY43T4DI77IJNNNNNS2VGJF6Q, seg_pos={0 0}, upload=429469]: Done uploading piece. err: <nil> Dec 26 00:47:16.722549123 (489h18m16.593632168s) [seg_pos={0 0}, upload=429469]: Segment reached optimal threshold of 13 pieces. Dec 26 00:47:16.722762218 (489h18m16.593845253s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:40104->77.90.31.18:4805: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 00:47:16.722770584 (489h18m16.593853629s) [noise=true , address=77.90.31.18:4805 , piece_id=3JDLPXPXUCQLCY5GGFIO5ALXQUIV2ZPFQU7P7JV74TIX72QW7HEQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.722809478 (489h18m16.593892513s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43588->77.90.31.15:4700: use of closed network connection Dec 26 00:47:16.722849113 (489h18m16.593932148s) [noise=true , address=77.90.31.15:4700 , piece_id=EP64GNU2PPOKZYPE3KAZTKT2NQ2GXBTRPKIHTN3IUTDI3RYGIBOQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723132743 (489h18m16.594215798s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj): protocol: expected piece hash; context canceled; EOF Dec 26 00:47:16.723142092 (489h18m16.594225127s) [noise=true , address=145.79.227.51:4409 , piece_id=IGZULJ2YZFUO44MIIG3NBH5OISGAMHQXJKYDQQJHRLEWVNQIKXPQ, seg_pos={0 0}, upload=429469]: Long tail cancellation due to optimal threshold Dec 26 00:47:16.723162931 (489h18m16.594245976s) [seg_pos={0 0}, upload=429469]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:47:16.723303749 (489h18m16.594386794s) [upload=429469]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:47:16.723311493 (489h18m16.594394528s) [upload=429469]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:47:17.010910779 (489h18m16.881993824s) [upload=429469]: Done uploading segments... Dec 26 00:47:17.010932401 (489h18m16.882015436s) [upload=429469]: Upload finished. err: <nil> Duration: 30.002322428s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................X.....X......X...X....X] duration: 30.002453396s trace: https://jaeger.dev.storj.io/trace/4c7382785cfee5e2 FAIL (2025-12-26 00:47:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: EVBVHANWNSBBKIF6JXWWZ5ASUTO42EU2LEW4V4OTKKOKU4I42VFQ) context deadline exceeded; piecestore: (Node ID: 1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, Piece ID: HHQ6LVYECU4MSN3DLU3ZGADVTMOGXMPZV664JTGYQ6V4RUI5WLEA) context deadline exceeded ======LOGS====== Dec 26 00:46:43.362873209 (489h17m43.233956244s) [upload=429460]: Starting upload Dec 26 00:46:43.362941690 (489h17m43.234024725s) [upload=429460]: Done starting upload Dec 26 00:46:43.363045437 (489h17m43.234128472s) [upload=429460]: Uploading segments... Dec 26 00:46:43.363145027 (489h17m43.234228072s) [upload=429460]: Got next segment. Inline: false Dec 26 00:46:43.363180945 (489h17m43.23426398s) [upload=429460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:46:43.363186926 (489h17m43.234269972s) [upload=429460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:46:43.614475365 (489h17m43.4855584s) [seg_pos={0 0}, upload=429460]: Begin upload segment... Dec 26 00:46:43.614529318 (489h17m43.485612363s) [seg_pos={0 0}, upload=429460]: Done begin upload segment. Dec 26 00:46:43.614554346 (489h17m43.485637391s) [noise=true , address=145.79.227.51:4409 , piece_id=JMMIBCQBUUI65ZP32EJNO53OBWAWZXM43TRKPF23H375KEMD3MRA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.614581758 (489h17m43.485664793s) [noise=true , address=145.79.227.19:4079 , piece_id=H7UKYP6TCQI22W4WO4YWJQ7S3USXJ5LFLKT5OJRLN2XINLFF36SA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.614892870 (489h17m43.485975905s) [noise=true , address=145.79.227.37:4567 , piece_id=ITUDX2EYDIHDVFCVCVR5IENOYINLQEBN2EVISZPNKEESWP5CO5UQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.615212699 (489h17m43.486295744s) [noise=true , address=45.91.116.11:4117 , piece_id=7CC5PF2F3AI7XQ3TRLJPSE7NZRTRJWGPLHTPFRRADSC7SD7MWDZQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.615274908 (489h17m43.486357943s) [noise=true , address=145.79.227.38:4979 , piece_id=WD437I22NZRLD2CP2O7JISP7ULDOKMNO3XBZFV4D2FGQHOEWANZA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.615910047 (489h17m43.486993082s) [noise=true , address=145.79.227.11:4730 , piece_id=XYGECZP5A3UUC3HBPYXG2KBVFW2L5N3FXEFIN5FUE4IH7A5ERSIQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.616257929 (489h17m43.487340965s) [noise=true , address=145.79.227.16:4910 , piece_id=EVBVHANWNSBBKIF6JXWWZ5ASUTO42EU2LEW4V4OTKKOKU4I42VFQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.616375293 (489h17m43.487458338s) [noise=true , address=145.79.227.47:4267 , piece_id=Q27ZOQ2A6QLUCZHH7AWAKMP3KZTCDFVKJ6RZMG7RF74UTCCV3LQQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.616624046 (489h17m43.487707081s) [noise=true , address=145.79.227.3:4149 , piece_id=O6PRJK6GIXNNTTXYYKG4RBLSJ7SLIUPUBKL5SXBRGC5GUBVTDLDA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.617461942 (489h17m43.488544977s) [noise=true , address=145.79.227.32:4834 , piece_id=HHQ6LVYECU4MSN3DLU3ZGADVTMOGXMPZV664JTGYQ6V4RUI5WLEA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.618121588 (489h17m43.489204623s) [noise=true , address=145.79.227.30:4761 , piece_id=WLZZXBZ4VFELNEZ6UYMA4CQ7L2LBZTOPPRIAT4RFBOXMYHPB6ZXA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.618720448 (489h17m43.489803493s) [noise=true , address=45.91.116.14:4192 , piece_id=7FSP23UKUEA3AOTM22IWEBZCEDLWRZGTCD46PMOXI2M3MIUEFDUQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.619132513 (489h17m43.490215548s) [noise=true , address=145.79.227.41:4085 , piece_id=AP5ALR2SXIYRGOXT4UMCZE4JAKRVYR5M37QVDHL5MRLMXY5GPA2A, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.620402220 (489h17m43.491485255s) [noise=true , address=145.79.227.17:4971 , piece_id=RT45XVRY55XTVLYIWQ73IVELNN4DINV6H3UYZLTPST2LX3IYYXOQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.621599130 (489h17m43.492682165s) [noise=true , address=145.79.227.42:4123 , piece_id=C6D7CHMUCP3XADXXEWASQZI22RZVKVJ4ITZJW43L24JL5FEIOLSA, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:43.622823922 (489h17m43.493906957s) [noise=true , address=145.79.227.34:4907 , piece_id=XZLNQGGQSBTR2H6MMQFSLYHUBBA65YDQJSXP3NWPC64WYDK52IHQ, seg_pos={0 0}, upload=429460]: Uploading piece... Dec 26 00:46:45.235222520 (489h17m45.106305565s) [upload=429460]: Next returned nil segment Dec 26 00:46:45.235240014 (489h17m45.106323049s) [upload=429460]: Waiting for error group managing segments... Dec 26 00:46:45.621712608 (489h17m45.492795643s) [noise=true , address=145.79.227.51:4409 , piece_id=JMMIBCQBUUI65ZP32EJNO53OBWAWZXM43TRKPF23H375KEMD3MRA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.627734554 (489h17m45.498817589s) [noise=true , address=145.79.227.11:4730 , piece_id=XYGECZP5A3UUC3HBPYXG2KBVFW2L5N3FXEFIN5FUE4IH7A5ERSIQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.637777616 (489h17m45.508860651s) [noise=true , address=145.79.227.19:4079 , piece_id=H7UKYP6TCQI22W4WO4YWJQ7S3USXJ5LFLKT5OJRLN2XINLFF36SA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.638744517 (489h17m45.509827562s) [noise=true , address=45.91.116.11:4117 , piece_id=7CC5PF2F3AI7XQ3TRLJPSE7NZRTRJWGPLHTPFRRADSC7SD7MWDZQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.639613271 (489h17m45.510696307s) [noise=true , address=145.79.227.34:4907 , piece_id=XZLNQGGQSBTR2H6MMQFSLYHUBBA65YDQJSXP3NWPC64WYDK52IHQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.646038124 (489h17m45.517121159s) [noise=true , address=145.79.227.42:4123 , piece_id=C6D7CHMUCP3XADXXEWASQZI22RZVKVJ4ITZJW43L24JL5FEIOLSA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.649072642 (489h17m45.520155687s) [noise=true , address=145.79.227.38:4979 , piece_id=WD437I22NZRLD2CP2O7JISP7ULDOKMNO3XBZFV4D2FGQHOEWANZA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.653998952 (489h17m45.525081987s) [noise=true , address=145.79.227.41:4085 , piece_id=AP5ALR2SXIYRGOXT4UMCZE4JAKRVYR5M37QVDHL5MRLMXY5GPA2A, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.657741318 (489h17m45.528824353s) [noise=true , address=145.79.227.37:4567 , piece_id=ITUDX2EYDIHDVFCVCVR5IENOYINLQEBN2EVISZPNKEESWP5CO5UQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.658893592 (489h17m45.529976627s) [noise=true , address=145.79.227.47:4267 , piece_id=Q27ZOQ2A6QLUCZHH7AWAKMP3KZTCDFVKJ6RZMG7RF74UTCCV3LQQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.660660976 (489h17m45.531744011s) [noise=true , address=145.79.227.32:4834 , piece_id=HHQ6LVYECU4MSN3DLU3ZGADVTMOGXMPZV664JTGYQ6V4RUI5WLEA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.664591349 (489h17m45.535674395s) [noise=true , address=45.91.116.14:4192 , piece_id=7FSP23UKUEA3AOTM22IWEBZCEDLWRZGTCD46PMOXI2M3MIUEFDUQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.682569911 (489h17m45.553652946s) [noise=true , address=145.79.227.16:4910 , piece_id=EVBVHANWNSBBKIF6JXWWZ5ASUTO42EU2LEW4V4OTKKOKU4I42VFQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.682590040 (489h17m45.553673075s) [seg_pos={0 0}, upload=429460]: Segment reached optimal threshold of 13 pieces. Dec 26 00:46:45.682726159 (489h17m45.553809194s) [noise=true , address=145.79.227.17:4971 , piece_id=RT45XVRY55XTVLYIWQ73IVELNN4DINV6H3UYZLTPST2LX3IYYXOQ, seg_pos={0 0}, upload=429460]: Done uploading piece. err: <nil> Dec 26 00:46:45.682781544 (489h17m45.553864589s) [noise=true , address=145.79.227.30:4761 , piece_id=WLZZXBZ4VFELNEZ6UYMA4CQ7L2LBZTOPPRIAT4RFBOXMYHPB6ZXA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF Dec 26 00:46:45.682788949 (489h17m45.553871984s) [noise=true , address=145.79.227.30:4761 , piece_id=WLZZXBZ4VFELNEZ6UYMA4CQ7L2LBZTOPPRIAT4RFBOXMYHPB6ZXA, seg_pos={0 0}, upload=429460]: Long tail cancellation due to optimal threshold Dec 26 00:46:45.682902375 (489h17m45.55398542s) [noise=true , address=145.79.227.3:4149 , piece_id=O6PRJK6GIXNNTTXYYKG4RBLSJ7SLIUPUBKL5SXBRGC5GUBVTDLDA, seg_pos={0 0}, upload=429460]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm): protocol: expected piece hash; context canceled; EOF Dec 26 00:46:45.682907955 (489h17m45.55399099s) [noise=true , address=145.79.227.3:4149 , piece_id=O6PRJK6GIXNNTTXYYKG4RBLSJ7SLIUPUBKL5SXBRGC5GUBVTDLDA, seg_pos={0 0}, upload=429460]: Long tail cancellation due to optimal threshold Dec 26 00:46:45.682922382 (489h17m45.554005417s) [seg_pos={0 0}, upload=429460]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 26 00:46:45.682989099 (489h17m45.554072135s) [upload=429460]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:46:45.682993317 (489h17m45.554076352s) [upload=429460]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:46:45.996208888 (489h17m45.867291923s) [upload=429460]: Done uploading segments... Dec 26 00:46:45.996233685 (489h17m45.86731672s) [upload=429460]: Upload finished. err: <nil> Duration: 30.002472072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.....X......X...X....] duration: 5.704493963s trace: https://jaeger.dev.storj.io/trace/2519ea020322b413 FAIL (2025-12-26 00:46:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 00:43:57.085881080 (489h14m56.956964115s) [upload=429415]: Got next segment. Inline: false Dec 26 00:43:57.085911478 (489h14m56.956994513s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:43:57.085915967 (489h14m56.956999002s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:43:57.662242656 (489h14m57.533325702s) [seg_pos={0 0}, upload=429415]: Begin upload segment... Dec 26 00:43:57.662377343 (489h14m57.533460388s) [seg_pos={0 0}, upload=429415]: Done begin upload segment. Dec 26 00:43:57.662438540 (489h14m57.533521585s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.662476141 (489h14m57.533559176s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.663264812 (489h14m57.534347848s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664124239 (489h14m57.535207284s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664164987 (489h14m57.535248022s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664787893 (489h14m57.535870928s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665089967 (489h14m57.536173013s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665766536 (489h14m57.536849571s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.666272159 (489h14m57.537355194s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.667115424 (489h14m57.538198469s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672164218 (489h14m57.543247263s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672852698 (489h14m57.543935743s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.673935921 (489h14m57.545018956s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.675858591 (489h14m57.546941636s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.677377683 (489h14m57.548460718s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.679011414 (489h14m57.550094469s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:59.470803103 (489h14m59.341886138s) [upload=429415]: Next returned nil segment Dec 26 00:43:59.470816949 (489h14m59.341899984s) [upload=429415]: Waiting for error group managing segments... Dec 26 00:43:59.823476783 (489h14m59.694559818s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.831469320 (489h14m59.702552355s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.846982108 (489h14m59.718065143s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.849323756 (489h14m59.720406791s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.852086416 (489h14m59.723169452s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.858575121 (489h14m59.729658166s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.863764491 (489h14m59.734847526s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.864365726 (489h14m59.735448762s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.867518640 (489h14m59.738601675s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.868318672 (489h14m59.739401717s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.880599135 (489h14m59.75168217s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.896814099 (489h14m59.767897134s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897604042 (489h14m59.768687117s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897622107 (489h14m59.768705152s) [seg_pos={0 0}, upload=429415]: Segment reached optimal threshold of 13 pieces. Dec 26 00:43:59.897840062 (489h14m59.768923097s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection Dec 26 00:43:59.897848879 (489h14m59.768931914s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.897861583 (489h14m59.768944618s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection Dec 26 00:43:59.897865861 (489h14m59.768948896s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898013162 (489h14m59.769096197s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 00:43:59.898022169 (489h14m59.769105204s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898133540 (489h14m59.769216575s) [seg_pos={0 0}, upload=429415]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:43:59.898233741 (489h14m59.769316786s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:43:59.898241276 (489h14m59.769324311s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:44:00.532776929 (489h15m0.403859964s) [upload=429415]: Done uploading segments... Dec 26 00:44:00.532796236 (489h15m0.403879271s) [upload=429415]: Upload finished. err: <nil> Duration: 5.70451915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.....X......X...X...] duration: 5.745231467s trace: https://jaeger.dev.storj.io/trace/2519ea020322b413 FAIL (2025-12-26 00:45:43 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 00:43:57.085881080 (489h14m56.956964115s) [upload=429415]: Got next segment. Inline: false Dec 26 00:43:57.085911478 (489h14m56.956994513s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:43:57.085915967 (489h14m56.956999002s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:43:57.662242656 (489h14m57.533325702s) [seg_pos={0 0}, upload=429415]: Begin upload segment... Dec 26 00:43:57.662377343 (489h14m57.533460388s) [seg_pos={0 0}, upload=429415]: Done begin upload segment. Dec 26 00:43:57.662438540 (489h14m57.533521585s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.662476141 (489h14m57.533559176s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.663264812 (489h14m57.534347848s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664124239 (489h14m57.535207284s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664164987 (489h14m57.535248022s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664787893 (489h14m57.535870928s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665089967 (489h14m57.536173013s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665766536 (489h14m57.536849571s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.666272159 (489h14m57.537355194s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.667115424 (489h14m57.538198469s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672164218 (489h14m57.543247263s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672852698 (489h14m57.543935743s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.673935921 (489h14m57.545018956s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.675858591 (489h14m57.546941636s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.677377683 (489h14m57.548460718s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.679011414 (489h14m57.550094469s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:59.470803103 (489h14m59.341886138s) [upload=429415]: Next returned nil segment Dec 26 00:43:59.470816949 (489h14m59.341899984s) [upload=429415]: Waiting for error group managing segments... Dec 26 00:43:59.823476783 (489h14m59.694559818s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.831469320 (489h14m59.702552355s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.846982108 (489h14m59.718065143s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.849323756 (489h14m59.720406791s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.852086416 (489h14m59.723169452s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.858575121 (489h14m59.729658166s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.863764491 (489h14m59.734847526s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.864365726 (489h14m59.735448762s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.867518640 (489h14m59.738601675s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.868318672 (489h14m59.739401717s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.880599135 (489h14m59.75168217s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.896814099 (489h14m59.767897134s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897604042 (489h14m59.768687117s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897622107 (489h14m59.768705152s) [seg_pos={0 0}, upload=429415]: Segment reached optimal threshold of 13 pieces. Dec 26 00:43:59.897840062 (489h14m59.768923097s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection Dec 26 00:43:59.897848879 (489h14m59.768931914s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.897861583 (489h14m59.768944618s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection Dec 26 00:43:59.897865861 (489h14m59.768948896s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898013162 (489h14m59.769096197s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 00:43:59.898022169 (489h14m59.769105204s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898133540 (489h14m59.769216575s) [seg_pos={0 0}, upload=429415]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:43:59.898233741 (489h14m59.769316786s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:43:59.898241276 (489h14m59.769324311s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:44:00.532776929 (489h15m0.403859964s) [upload=429415]: Done uploading segments... Dec 26 00:44:00.532796236 (489h15m0.403879271s) [upload=429415]: Upload finished. err: <nil> Duration: 5.745249432s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.....X......X...X..] duration: 5.735519526s trace: https://jaeger.dev.storj.io/trace/2519ea020322b413 FAIL (2025-12-26 00:45:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 00:43:57.085881080 (489h14m56.956964115s) [upload=429415]: Got next segment. Inline: false Dec 26 00:43:57.085911478 (489h14m56.956994513s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:43:57.085915967 (489h14m56.956999002s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:43:57.662242656 (489h14m57.533325702s) [seg_pos={0 0}, upload=429415]: Begin upload segment... Dec 26 00:43:57.662377343 (489h14m57.533460388s) [seg_pos={0 0}, upload=429415]: Done begin upload segment. Dec 26 00:43:57.662438540 (489h14m57.533521585s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.662476141 (489h14m57.533559176s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.663264812 (489h14m57.534347848s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664124239 (489h14m57.535207284s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664164987 (489h14m57.535248022s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664787893 (489h14m57.535870928s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665089967 (489h14m57.536173013s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665766536 (489h14m57.536849571s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.666272159 (489h14m57.537355194s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.667115424 (489h14m57.538198469s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672164218 (489h14m57.543247263s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672852698 (489h14m57.543935743s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.673935921 (489h14m57.545018956s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.675858591 (489h14m57.546941636s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.677377683 (489h14m57.548460718s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.679011414 (489h14m57.550094469s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:59.470803103 (489h14m59.341886138s) [upload=429415]: Next returned nil segment Dec 26 00:43:59.470816949 (489h14m59.341899984s) [upload=429415]: Waiting for error group managing segments... Dec 26 00:43:59.823476783 (489h14m59.694559818s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.831469320 (489h14m59.702552355s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.846982108 (489h14m59.718065143s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.849323756 (489h14m59.720406791s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.852086416 (489h14m59.723169452s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.858575121 (489h14m59.729658166s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.863764491 (489h14m59.734847526s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.864365726 (489h14m59.735448762s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.867518640 (489h14m59.738601675s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.868318672 (489h14m59.739401717s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.880599135 (489h14m59.75168217s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.896814099 (489h14m59.767897134s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897604042 (489h14m59.768687117s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897622107 (489h14m59.768705152s) [seg_pos={0 0}, upload=429415]: Segment reached optimal threshold of 13 pieces. Dec 26 00:43:59.897840062 (489h14m59.768923097s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection Dec 26 00:43:59.897848879 (489h14m59.768931914s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.897861583 (489h14m59.768944618s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection Dec 26 00:43:59.897865861 (489h14m59.768948896s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898013162 (489h14m59.769096197s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 00:43:59.898022169 (489h14m59.769105204s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898133540 (489h14m59.769216575s) [seg_pos={0 0}, upload=429415]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:43:59.898233741 (489h14m59.769316786s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:43:59.898241276 (489h14m59.769324311s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:44:00.532776929 (489h15m0.403859964s) [upload=429415]: Done uploading segments... Dec 26 00:44:00.532796236 (489h15m0.403879271s) [upload=429415]: Upload finished. err: <nil> Duration: 5.735545806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.....X......X...X.] duration: 5.753682431s trace: https://jaeger.dev.storj.io/trace/2519ea020322b413 FAIL (2025-12-26 00:44:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 00:43:57.085881080 (489h14m56.956964115s) [upload=429415]: Got next segment. Inline: false Dec 26 00:43:57.085911478 (489h14m56.956994513s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:43:57.085915967 (489h14m56.956999002s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:43:57.662242656 (489h14m57.533325702s) [seg_pos={0 0}, upload=429415]: Begin upload segment... Dec 26 00:43:57.662377343 (489h14m57.533460388s) [seg_pos={0 0}, upload=429415]: Done begin upload segment. Dec 26 00:43:57.662438540 (489h14m57.533521585s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.662476141 (489h14m57.533559176s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.663264812 (489h14m57.534347848s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664124239 (489h14m57.535207284s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664164987 (489h14m57.535248022s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664787893 (489h14m57.535870928s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665089967 (489h14m57.536173013s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665766536 (489h14m57.536849571s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.666272159 (489h14m57.537355194s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.667115424 (489h14m57.538198469s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672164218 (489h14m57.543247263s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672852698 (489h14m57.543935743s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.673935921 (489h14m57.545018956s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.675858591 (489h14m57.546941636s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.677377683 (489h14m57.548460718s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.679011414 (489h14m57.550094469s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:59.470803103 (489h14m59.341886138s) [upload=429415]: Next returned nil segment Dec 26 00:43:59.470816949 (489h14m59.341899984s) [upload=429415]: Waiting for error group managing segments... Dec 26 00:43:59.823476783 (489h14m59.694559818s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.831469320 (489h14m59.702552355s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.846982108 (489h14m59.718065143s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.849323756 (489h14m59.720406791s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.852086416 (489h14m59.723169452s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.858575121 (489h14m59.729658166s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.863764491 (489h14m59.734847526s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.864365726 (489h14m59.735448762s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.867518640 (489h14m59.738601675s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.868318672 (489h14m59.739401717s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.880599135 (489h14m59.75168217s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.896814099 (489h14m59.767897134s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897604042 (489h14m59.768687117s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897622107 (489h14m59.768705152s) [seg_pos={0 0}, upload=429415]: Segment reached optimal threshold of 13 pieces. Dec 26 00:43:59.897840062 (489h14m59.768923097s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection Dec 26 00:43:59.897848879 (489h14m59.768931914s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.897861583 (489h14m59.768944618s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection Dec 26 00:43:59.897865861 (489h14m59.768948896s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898013162 (489h14m59.769096197s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 00:43:59.898022169 (489h14m59.769105204s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898133540 (489h14m59.769216575s) [seg_pos={0 0}, upload=429415]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:43:59.898233741 (489h14m59.769316786s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:43:59.898241276 (489h14m59.769324311s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:44:00.532776929 (489h15m0.403859964s) [upload=429415]: Done uploading segments... Dec 26 00:44:00.532796236 (489h15m0.403879271s) [upload=429415]: Upload finished. err: <nil> Duration: 5.753713049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.....X......X...X] duration: 30.00322234s trace: https://jaeger.dev.storj.io/trace/2519ea020322b413 FAIL (2025-12-26 00:44:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ts... Dec 26 00:43:57.085881080 (489h14m56.956964115s) [upload=429415]: Got next segment. Inline: false Dec 26 00:43:57.085911478 (489h14m56.956994513s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:43:57.085915967 (489h14m56.956999002s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:43:57.662242656 (489h14m57.533325702s) [seg_pos={0 0}, upload=429415]: Begin upload segment... Dec 26 00:43:57.662377343 (489h14m57.533460388s) [seg_pos={0 0}, upload=429415]: Done begin upload segment. Dec 26 00:43:57.662438540 (489h14m57.533521585s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.662476141 (489h14m57.533559176s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.663264812 (489h14m57.534347848s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664124239 (489h14m57.535207284s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664164987 (489h14m57.535248022s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.664787893 (489h14m57.535870928s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665089967 (489h14m57.536173013s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.665766536 (489h14m57.536849571s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.666272159 (489h14m57.537355194s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.667115424 (489h14m57.538198469s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672164218 (489h14m57.543247263s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.672852698 (489h14m57.543935743s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.673935921 (489h14m57.545018956s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.675858591 (489h14m57.546941636s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.677377683 (489h14m57.548460718s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:57.679011414 (489h14m57.550094469s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Uploading piece... Dec 26 00:43:59.470803103 (489h14m59.341886138s) [upload=429415]: Next returned nil segment Dec 26 00:43:59.470816949 (489h14m59.341899984s) [upload=429415]: Waiting for error group managing segments... Dec 26 00:43:59.823476783 (489h14m59.694559818s) [noise=true , address=77.90.31.10:4518 , piece_id=CJTTGZ2XEJSARREC3LCEQ4WK62H2VYGYDHVHZGFJKJFWDP5F4UQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.831469320 (489h14m59.702552355s) [noise=true , address=145.79.227.12:4765 , piece_id=3FUH7EV6NKE2KPQPXDTCVJJSWLGUVPNZFMMSBVACKDTWAVJO6YEQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.846982108 (489h14m59.718065143s) [noise=true , address=45.91.116.13:4155 , piece_id=R5MCI6ZAYWCC4HIV7WTFI3UXBFU255YAEUWCHZS2NWNHMZDY4EZQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.849323756 (489h14m59.720406791s) [noise=true , address=145.79.227.42:4121 , piece_id=5KJCK23GZ3AOGPQ6YZQDAE7DWEDG2HXXLGY4AVB7RTNGAJPGVEZA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.852086416 (489h14m59.723169452s) [noise=true , address=145.79.227.35:4482 , piece_id=OBAMFV7CWJDKZYF4W3O7MIH35S5JCGH2SVA2LFCESU5EQJUOVBCA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.858575121 (489h14m59.729658166s) [noise=true , address=45.91.116.5:4300 , piece_id=4PHAFJR52X2LXOR7KRY6VPVGZCNTOH2KC2AHKV5WRHC65C3WNYFQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.863764491 (489h14m59.734847526s) [noise=true , address=45.91.116.11:4117 , piece_id=54DZABLZNZDPNHFYRDA2IDXUO4OQ6A7MMJ6YMB23SCKTGBMUPMGA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.864365726 (489h14m59.735448762s) [noise=true , address=145.79.227.39:4013 , piece_id=RT2ZOFX35JVKT3JIM2UE5B6S4A4ABR6DQOOQG66DKS7BF2VCG3PQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.867518640 (489h14m59.738601675s) [noise=true , address=145.79.227.49:4339 , piece_id=QIS2MHBQKLCL56DQAYKUUTA4VVU7RYA76VLR3E2XIVXARGV2AXIA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.868318672 (489h14m59.739401717s) [noise=true , address=145.79.227.47:4266 , piece_id=U72RJ4WZ3EPSGMT5GLEC2G3OLQUYJNTRJCLEZEZNSTBGBY772RLA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.880599135 (489h14m59.75168217s) [noise=true , address=77.90.31.16:4736 , piece_id=RIYTEX6COWTCCJV5R6UPXZWSZOTYOAI4X7YBCK654GXSC3DHKOPA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.896814099 (489h14m59.767897134s) [noise=true , address=45.91.116.6:4975 , piece_id=B7BMLUWQRDRGG2QUL5BE2JQ77B7TB3XU2AXXBM2DHZYJLXT4VJ3A, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897604042 (489h14m59.768687117s) [noise=true , address=145.79.227.13:4804 , piece_id=F3DD2D4K47CVVX2L2ZN2ZFOFQ3SVDLBXNCHYRYGQSKI6WYV5KHUA, seg_pos={0 0}, upload=429415]: Done uploading piece. err: <nil> Dec 26 00:43:59.897622107 (489h14m59.768705152s) [seg_pos={0 0}, upload=429415]: Segment reached optimal threshold of 13 pieces. Dec 26 00:43:59.897840062 (489h14m59.768923097s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma): protocol: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48868->145.79.227.38:4979: use of closed network connection Dec 26 00:43:59.897848879 (489h14m59.768931914s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6SV7R6T5CCEN4A2HLBVRP5OJAOOWQM6ZXI4G43KGIHUVS56KTWQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.897861583 (489h14m59.768944618s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58830->77.90.31.14:4662: use of closed network connection Dec 26 00:43:59.897865861 (489h14m59.768948896s) [noise=true , address=77.90.31.14:4662 , piece_id=IRQE5LC24HKKFGENB7SBELQFZ6FK3HQ7YNAPIDHZNWWGPRMSWKQQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898013162 (489h14m59.769096197s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 00:43:59.898022169 (489h14m59.769105204s) [noise=true , address=145.79.227.15:4876 , piece_id=LTKUKXHCSWW2GHMPAAUCKWKOSBRD2MWMI4QUX53D2NBV2DIXH5AQ, seg_pos={0 0}, upload=429415]: Long tail cancellation due to optimal threshold Dec 26 00:43:59.898133540 (489h14m59.769216575s) [seg_pos={0 0}, upload=429415]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:43:59.898233741 (489h14m59.769316786s) [upload=429415]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:43:59.898241276 (489h14m59.769324311s) [upload=429415]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:44:00.532776929 (489h15m0.403859964s) [upload=429415]: Done uploading segments... Dec 26 00:44:00.532796236 (489h15m0.403879271s) [upload=429415]: Upload finished. err: <nil> Duration: 30.003246264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.....X......X...] duration: 5.682604724s trace: https://jaeger.dev.storj.io/trace/284a3db52db9ad91 FAIL (2025-12-26 00:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 00:40:50.594608426 (489h11m50.465691471s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:40:50.594613456 (489h11m50.465696491s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:40:50.843877811 (489h11m50.714960857s) [seg_pos={0 0}, upload=429366]: Begin upload segment... Dec 26 00:40:50.843929030 (489h11m50.715012065s) [seg_pos={0 0}, upload=429366]: Done begin upload segment. Dec 26 00:40:50.843959567 (489h11m50.715042642s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.843975888 (489h11m50.715058933s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844515836 (489h11m50.715598881s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844862025 (489h11m50.71594507s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845134464 (489h11m50.716217499s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845392475 (489h11m50.71647552s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845696464 (489h11m50.716779499s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846074093 (489h11m50.717157129s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846483112 (489h11m50.717566157s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848115229 (489h11m50.719198275s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848235959 (489h11m50.719318994s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.849404555 (489h11m50.72048759s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.851266469 (489h11m50.722349505s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.852876715 (489h11m50.72395976s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.854067773 (489h11m50.725150818s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.857610809 (489h11m50.728693844s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:52.657107506 (489h11m52.528190551s) [upload=429366]: Next returned nil segment Dec 26 00:40:52.657132855 (489h11m52.52821591s) [upload=429366]: Waiting for error group managing segments... Dec 26 00:40:52.997250667 (489h11m52.868333702s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.002494862 (489h11m52.873577918s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.007405563 (489h11m52.878488598s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.015058013 (489h11m52.886141058s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.031072816 (489h11m52.902155851s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.040932919 (489h11m52.912015954s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.045152163 (489h11m52.916235208s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.056600761 (489h11m52.927683796s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.101395521 (489h11m52.972478556s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.108619316 (489h11m52.979702361s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.166246600 (489h11m53.037329645s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.256386058 (489h11m53.127469093s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325180885 (489h11m53.19626393s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325203879 (489h11m53.196286924s) [seg_pos={0 0}, upload=429366]: Segment reached optimal threshold of 13 pieces. Dec 26 00:40:53.325396605 (489h11m53.19647965s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection Dec 26 00:40:53.325404010 (489h11m53.196487055s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.325717286 (489h11m53.196800331s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection Dec 26 00:40:53.325724851 (489h11m53.196807886s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326110024 (489h11m53.197193079s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:53506->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 00:40:53.326119462 (489h11m53.197202497s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326139581 (489h11m53.197222616s) [seg_pos={0 0}, upload=429366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:40:53.326213642 (489h11m53.197296677s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:40:53.326221677 (489h11m53.197304722s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:40:53.616215271 (489h11m53.487298306s) [upload=429366]: Done uploading segments... Dec 26 00:40:53.616238846 (489h11m53.487321921s) [upload=429366]: Upload finished. err: <nil> Duration: 5.682633488s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.....X......X..] duration: 5.832720461s trace: https://jaeger.dev.storj.io/trace/284a3db52db9ad91 FAIL (2025-12-26 00:42:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 00:40:50.594608426 (489h11m50.465691471s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:40:50.594613456 (489h11m50.465696491s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:40:50.843877811 (489h11m50.714960857s) [seg_pos={0 0}, upload=429366]: Begin upload segment... Dec 26 00:40:50.843929030 (489h11m50.715012065s) [seg_pos={0 0}, upload=429366]: Done begin upload segment. Dec 26 00:40:50.843959567 (489h11m50.715042642s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.843975888 (489h11m50.715058933s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844515836 (489h11m50.715598881s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844862025 (489h11m50.71594507s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845134464 (489h11m50.716217499s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845392475 (489h11m50.71647552s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845696464 (489h11m50.716779499s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846074093 (489h11m50.717157129s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846483112 (489h11m50.717566157s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848115229 (489h11m50.719198275s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848235959 (489h11m50.719318994s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.849404555 (489h11m50.72048759s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.851266469 (489h11m50.722349505s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.852876715 (489h11m50.72395976s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.854067773 (489h11m50.725150818s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.857610809 (489h11m50.728693844s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:52.657107506 (489h11m52.528190551s) [upload=429366]: Next returned nil segment Dec 26 00:40:52.657132855 (489h11m52.52821591s) [upload=429366]: Waiting for error group managing segments... Dec 26 00:40:52.997250667 (489h11m52.868333702s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.002494862 (489h11m52.873577918s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.007405563 (489h11m52.878488598s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.015058013 (489h11m52.886141058s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.031072816 (489h11m52.902155851s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.040932919 (489h11m52.912015954s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.045152163 (489h11m52.916235208s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.056600761 (489h11m52.927683796s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.101395521 (489h11m52.972478556s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.108619316 (489h11m52.979702361s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.166246600 (489h11m53.037329645s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.256386058 (489h11m53.127469093s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325180885 (489h11m53.19626393s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325203879 (489h11m53.196286924s) [seg_pos={0 0}, upload=429366]: Segment reached optimal threshold of 13 pieces. Dec 26 00:40:53.325396605 (489h11m53.19647965s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection Dec 26 00:40:53.325404010 (489h11m53.196487055s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.325717286 (489h11m53.196800331s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection Dec 26 00:40:53.325724851 (489h11m53.196807886s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326110024 (489h11m53.197193079s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:53506->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 00:40:53.326119462 (489h11m53.197202497s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326139581 (489h11m53.197222616s) [seg_pos={0 0}, upload=429366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:40:53.326213642 (489h11m53.197296677s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:40:53.326221677 (489h11m53.197304722s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:40:53.616215271 (489h11m53.487298306s) [upload=429366]: Done uploading segments... Dec 26 00:40:53.616238846 (489h11m53.487321921s) [upload=429366]: Upload finished. err: <nil> Duration: 5.832749095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.....X......X.] duration: 28.085046583s trace: https://jaeger.dev.storj.io/trace/284a3db52db9ad91 FAIL (2025-12-26 00:41:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 00:40:50.594608426 (489h11m50.465691471s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:40:50.594613456 (489h11m50.465696491s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:40:50.843877811 (489h11m50.714960857s) [seg_pos={0 0}, upload=429366]: Begin upload segment... Dec 26 00:40:50.843929030 (489h11m50.715012065s) [seg_pos={0 0}, upload=429366]: Done begin upload segment. Dec 26 00:40:50.843959567 (489h11m50.715042642s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.843975888 (489h11m50.715058933s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844515836 (489h11m50.715598881s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844862025 (489h11m50.71594507s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845134464 (489h11m50.716217499s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845392475 (489h11m50.71647552s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845696464 (489h11m50.716779499s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846074093 (489h11m50.717157129s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846483112 (489h11m50.717566157s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848115229 (489h11m50.719198275s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848235959 (489h11m50.719318994s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.849404555 (489h11m50.72048759s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.851266469 (489h11m50.722349505s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.852876715 (489h11m50.72395976s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.854067773 (489h11m50.725150818s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.857610809 (489h11m50.728693844s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:52.657107506 (489h11m52.528190551s) [upload=429366]: Next returned nil segment Dec 26 00:40:52.657132855 (489h11m52.52821591s) [upload=429366]: Waiting for error group managing segments... Dec 26 00:40:52.997250667 (489h11m52.868333702s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.002494862 (489h11m52.873577918s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.007405563 (489h11m52.878488598s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.015058013 (489h11m52.886141058s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.031072816 (489h11m52.902155851s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.040932919 (489h11m52.912015954s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.045152163 (489h11m52.916235208s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.056600761 (489h11m52.927683796s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.101395521 (489h11m52.972478556s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.108619316 (489h11m52.979702361s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.166246600 (489h11m53.037329645s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.256386058 (489h11m53.127469093s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325180885 (489h11m53.19626393s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325203879 (489h11m53.196286924s) [seg_pos={0 0}, upload=429366]: Segment reached optimal threshold of 13 pieces. Dec 26 00:40:53.325396605 (489h11m53.19647965s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection Dec 26 00:40:53.325404010 (489h11m53.196487055s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.325717286 (489h11m53.196800331s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection Dec 26 00:40:53.325724851 (489h11m53.196807886s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326110024 (489h11m53.197193079s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:53506->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 00:40:53.326119462 (489h11m53.197202497s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326139581 (489h11m53.197222616s) [seg_pos={0 0}, upload=429366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:40:53.326213642 (489h11m53.197296677s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:40:53.326221677 (489h11m53.197304722s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:40:53.616215271 (489h11m53.487298306s) [upload=429366]: Done uploading segments... Dec 26 00:40:53.616238846 (489h11m53.487321921s) [upload=429366]: Upload finished. err: <nil> Duration: 28.085063225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.....X......X] duration: 30.002238883s trace: https://jaeger.dev.storj.io/trace/284a3db52db9ad91 FAIL (2025-12-26 00:41:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 00:40:50.594608426 (489h11m50.465691471s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:40:50.594613456 (489h11m50.465696491s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:40:50.843877811 (489h11m50.714960857s) [seg_pos={0 0}, upload=429366]: Begin upload segment... Dec 26 00:40:50.843929030 (489h11m50.715012065s) [seg_pos={0 0}, upload=429366]: Done begin upload segment. Dec 26 00:40:50.843959567 (489h11m50.715042642s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.843975888 (489h11m50.715058933s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844515836 (489h11m50.715598881s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.844862025 (489h11m50.71594507s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845134464 (489h11m50.716217499s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845392475 (489h11m50.71647552s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.845696464 (489h11m50.716779499s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846074093 (489h11m50.717157129s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.846483112 (489h11m50.717566157s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848115229 (489h11m50.719198275s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.848235959 (489h11m50.719318994s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.849404555 (489h11m50.72048759s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.851266469 (489h11m50.722349505s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.852876715 (489h11m50.72395976s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.854067773 (489h11m50.725150818s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:50.857610809 (489h11m50.728693844s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Uploading piece... Dec 26 00:40:52.657107506 (489h11m52.528190551s) [upload=429366]: Next returned nil segment Dec 26 00:40:52.657132855 (489h11m52.52821591s) [upload=429366]: Waiting for error group managing segments... Dec 26 00:40:52.997250667 (489h11m52.868333702s) [noise=true , address=145.79.227.39:4013 , piece_id=FRXUV6NB5SC7UJOCLEV5ZV6TAF3OFZBFMYFBKL4DB4L5QJBN7WSQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.002494862 (489h11m52.873577918s) [noise=true , address=77.90.31.12:4589 , piece_id=OSNHOWVFVBUNL2544GWPQO6NLC57ITZQT5ZGAZQQ4O4YKUNV67BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.007405563 (489h11m52.878488598s) [noise=true , address=145.79.227.51:4409 , piece_id=T4CGOYY2HBCDV5QEOG4WV5HVTUDDV5LIDDS7YNSUKFODTE7BZTVA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.015058013 (489h11m52.886141058s) [noise=true , address=145.79.227.48:4302 , piece_id=DDZG7R4O7W3YV44BQUDRAAIN6L2UWVYXL7PBZ2SDEQDZF5WDXU2Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.031072816 (489h11m52.902155851s) [noise=true , address=145.79.227.25:4617 , piece_id=LECIVSWOW76ZKSJ4D4BSALUKSKL5VPE3QLOKER3JTXATWMNTO2BQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.040932919 (489h11m52.912015954s) [noise=true , address=145.79.227.15:4873 , piece_id=7POIOLASPLVTXNI3LOL7V6HKFW4TYFCVSEPNCJOM24ZM7DAXV4PQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.045152163 (489h11m52.916235208s) [noise=true , address=145.79.227.46:4603 , piece_id=LO3AO6LC2MSK3NXGRUPNG6A4H6XZQYK3XN6NPUVDXZHMS2VW3X4Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.056600761 (489h11m52.927683796s) [noise=true , address=45.91.116.8:4048 , piece_id=NSA43S3KH7CO7FBZD4T5GVNTPXRNIEDOV76WHMXMQMRFVSHWUUHA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.101395521 (489h11m52.972478556s) [noise=true , address=45.91.116.14:4192 , piece_id=XYFVXXEKNSQFTRTMHZZTVELDB2I4HNFXZIMWW47L46KO6KLDN4LQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.108619316 (489h11m52.979702361s) [noise=true , address=145.79.227.17:4971 , piece_id=HIKPF6M6EDXWF5CZLX7QFHDFGNVT5J7Y3ZRU3FGBNRGSKLQKIXAA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.166246600 (489h11m53.037329645s) [noise=true , address=145.79.227.14:4839 , piece_id=66SEJQGLRB2IN2BRSAAEQKFTVKM5LCZ7HDVWS4TDK72KOWRG4JHQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.256386058 (489h11m53.127469093s) [noise=true , address=77.90.31.11:4554 , piece_id=KR3D7UIAZ6JGMS6SCQ3VGIXLFB3BUDQGM7Y32QZRYSWAGIDJJT5Q, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325180885 (489h11m53.19626393s) [noise=true , address=145.79.227.50:4373 , piece_id=7DULHDVJOFQYX2I3K5NFR2SFBXLZHBFDGM4YOC7NNKMSQGH63RLQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: <nil> Dec 26 00:40:53.325203879 (489h11m53.196286924s) [seg_pos={0 0}, upload=429366]: Segment reached optimal threshold of 13 pieces. Dec 26 00:40:53.325396605 (489h11m53.19647965s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54850->145.79.227.49:4339: use of closed network connection Dec 26 00:40:53.325404010 (489h11m53.196487055s) [noise=true , address=145.79.227.49:4339 , piece_id=SFAF7GX5WZTNLXYUCK2QIPSO3GZDH6S7HVBLHG6UMOINZFCZ5TOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.325717286 (489h11m53.196800331s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49578->145.79.227.37:4567: use of closed network connection Dec 26 00:40:53.325724851 (489h11m53.196807886s) [noise=true , address=145.79.227.37:4567 , piece_id=YH34637PZNKUZN2GOKF3S6JUYH55VKDIWL5IO2D5D27M6BT6TXOQ, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326110024 (489h11m53.197193079s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:53506->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 00:40:53.326119462 (489h11m53.197202497s) [noise=true , address=45.91.116.10:4083 , piece_id=G3V3NVKICIOTGO6TH3JX6VELBNGXYHOYMWZLCQ7BJ4VG63TDXEWA, seg_pos={0 0}, upload=429366]: Long tail cancellation due to optimal threshold Dec 26 00:40:53.326139581 (489h11m53.197222616s) [seg_pos={0 0}, upload=429366]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:40:53.326213642 (489h11m53.197296677s) [upload=429366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:40:53.326221677 (489h11m53.197304722s) [upload=429366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:40:53.616215271 (489h11m53.487298306s) [upload=429366]: Done uploading segments... Dec 26 00:40:53.616238846 (489h11m53.487321921s) [upload=429366]: Upload finished. err: <nil> Duration: 30.002259101s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.....X......] duration: 5.702199763s trace: https://jaeger.dev.storj.io/trace/25bf79ff9b270f19 FAIL (2025-12-26 00:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, Piece ID: EABKC2BQXJLCWXY3QOXBHG4MJTQKE75FHN3DXGX2C2UHVIDMY4CQ) context deadline exceeded; piecestore: (Node ID: 12ttjngUh34tGJpCBgVCLfqrWwsdmd88XKWPrmaJrgkMu8htkKM, Piece ID: GGXCYAHVQOP365YZSW7BOFDTZYFFZ6TTKZBUVVIIWMN5AF6KTRDA) context deadline exceeded; piecestore: (Node ID: 12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, Piece ID: LL5TL5FMYY7CEIWQF3TMVHVTGVBE4BYTSQBMECN33N7LE3KFFY6A) context deadline exceeded ======LOGS====== Dec 26 00:35:26.801070852 (489h6m26.672153897s) [upload=429285]: Starting upload Dec 26 00:35:26.801126829 (489h6m26.672209864s) [upload=429285]: Done starting upload Dec 26 00:35:26.801238581 (489h6m26.672321616s) [upload=429285]: Uploading segments... Dec 26 00:35:26.801325197 (489h6m26.672408232s) [upload=429285]: Got next segment. Inline: false Dec 26 00:35:26.801356817 (489h6m26.672439862s) [upload=429285]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 00:35:26.801361856 (489h6m26.672444891s) [upload=429285]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 00:35:27.066061883 (489h6m26.937144918s) [seg_pos={0 0}, upload=429285]: Begin upload segment... Dec 26 00:35:27.066108793 (489h6m26.937191838s) [seg_pos={0 0}, upload=429285]: Done begin upload segment. Dec 26 00:35:27.066144230 (489h6m26.937227286s) [noise=true , address=145.79.227.51:4409 , piece_id=BRUMICPOT4MYZGRJV3FJLSFYR364TSV6SWAEFWT6LX5O5IOJBHAA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.066214514 (489h6m26.937297579s) [noise=true , address=145.79.227.47:4265 , piece_id=L7PDIGOR3FA2V2YRH7VDZT4P3PO35P7MR7QEZ34PC3HYHEBC64IA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.066743742 (489h6m26.937826777s) [noise=true , address=145.79.227.15:4876 , piece_id=HQMR6OYIDWU2IHWAOFK3CTRVA5MGO4PMFWHYO536KCLLR3A5VMYA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.067019507 (489h6m26.938102542s) [noise=true , address=145.79.227.35:4482 , piece_id=LL5TL5FMYY7CEIWQF3TMVHVTGVBE4BYTSQBMECN33N7LE3KFFY6A, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.067286244 (489h6m26.938369279s) [noise=true , address=45.91.116.6:4975 , piece_id=PDMDHWLSKVVJC3XTANRGQBDQMNLITQ7NC7JD53OMND4SULINK6LQ, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.067555127 (489h6m26.938638162s) [noise=true , address=145.79.227.32:4835 , piece_id=LVCRY6KH5FJEO6V4GT2NPKIKFHFIYSUYYFRKFBD65AT5YDT3HWUQ, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.067899532 (489h6m26.938982568s) [noise=true , address=145.79.227.11:4729 , piece_id=OPXINPPBIOK4PWDAJ4TXF3U3MJQY2DNEZVCOFA2KOWULLYLXVO2A, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.068210875 (489h6m26.93929395s) [noise=true , address=145.79.227.41:4087 , piece_id=EABKC2BQXJLCWXY3QOXBHG4MJTQKE75FHN3DXGX2C2UHVIDMY4CQ, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.068596940 (489h6m26.939679975s) [noise=true , address=145.79.227.42:4123 , piece_id=GGXCYAHVQOP365YZSW7BOFDTZYFFZ6TTKZBUVVIIWMN5AF6KTRDA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.070024358 (489h6m26.941107404s) [noise=true , address=145.79.227.37:4566 , piece_id=OZHV3IU5SQN7FLNAFHEP4G4326Z5P5D767GNLYU3KBMQ2YKOFSIA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.071805530 (489h6m26.942888565s) [noise=true , address=145.79.227.39:4013 , piece_id=BFUZMBVNECRRKWOB5JI7NV4E3UUDTLPRO3HIS643N74JET6PKBGA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.075103499 (489h6m26.946186544s) [noise=true , address=145.79.227.3:4149 , piece_id=PAA7Y5SFNXOIA74HFUMZK52SLR42E3TZUVZUDTLMA2ATMFEOOONA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.077302065 (489h6m26.9483851s) [noise=true , address=45.91.116.12:4227 , piece_id=5SVWMKVY5BSOUGAYHMX3OPR2YCGEMMU7VRRQQR5242YOOI54U23A, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.079218363 (489h6m26.950301408s) [noise=true , address=145.79.227.36:4519 , piece_id=HCNGKMFRNEV2P2QM6DNNS5LXHS3F5NYMFFRI4AO4ITH2ZVABHLTA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.080324480 (489h6m26.951407515s) [noise=true , address=145.79.227.49:4339 , piece_id=ULH4WS5LPARMLEHHG2D3BBHQ3GSWED4RBG4EKLCWYL6R7XUCOKDA, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:27.080594554 (489h6m26.951677599s) [noise=true , address=145.79.227.2:4114 , piece_id=5XLLEUOHSCNSUGWRLWU4DRWMBCIFOQ5NDXY2TABK2YDCMQDQ2ZDQ, seg_pos={0 0}, upload=429285]: Uploading piece... Dec 26 00:35:28.909509125 (489h6m28.78059217s) [upload=429285]: Next returned nil segment Dec 26 00:35:28.909525918 (489h6m28.780608963s) [upload=429285]: Waiting for error group managing segments... Dec 26 00:35:29.203225169 (489h6m29.074308214s) [noise=true , address=145.79.227.11:4729 , piece_id=OPXINPPBIOK4PWDAJ4TXF3U3MJQY2DNEZVCOFA2KOWULLYLXVO2A, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.229260551 (489h6m29.100343586s) [noise=true , address=45.91.116.12:4227 , piece_id=5SVWMKVY5BSOUGAYHMX3OPR2YCGEMMU7VRRQQR5242YOOI54U23A, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.241959560 (489h6m29.113042605s) [noise=true , address=145.79.227.41:4087 , piece_id=EABKC2BQXJLCWXY3QOXBHG4MJTQKE75FHN3DXGX2C2UHVIDMY4CQ, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.255881658 (489h6m29.126964693s) [noise=true , address=45.91.116.6:4975 , piece_id=PDMDHWLSKVVJC3XTANRGQBDQMNLITQ7NC7JD53OMND4SULINK6LQ, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.263534949 (489h6m29.134617984s) [noise=true , address=145.79.227.35:4482 , piece_id=LL5TL5FMYY7CEIWQF3TMVHVTGVBE4BYTSQBMECN33N7LE3KFFY6A, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.268968466 (489h6m29.140051501s) [noise=true , address=145.79.227.37:4566 , piece_id=OZHV3IU5SQN7FLNAFHEP4G4326Z5P5D767GNLYU3KBMQ2YKOFSIA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.277787387 (489h6m29.148870422s) [noise=true , address=145.79.227.49:4339 , piece_id=ULH4WS5LPARMLEHHG2D3BBHQ3GSWED4RBG4EKLCWYL6R7XUCOKDA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.287675765 (489h6m29.1587588s) [noise=true , address=145.79.227.3:4149 , piece_id=PAA7Y5SFNXOIA74HFUMZK52SLR42E3TZUVZUDTLMA2ATMFEOOONA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.295254024 (489h6m29.166337059s) [noise=true , address=145.79.227.36:4519 , piece_id=HCNGKMFRNEV2P2QM6DNNS5LXHS3F5NYMFFRI4AO4ITH2ZVABHLTA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.542201629 (489h6m29.413284664s) [noise=true , address=145.79.227.32:4835 , piece_id=LVCRY6KH5FJEO6V4GT2NPKIKFHFIYSUYYFRKFBD65AT5YDT3HWUQ, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.557274718 (489h6m29.428357753s) [noise=true , address=145.79.227.42:4123 , piece_id=GGXCYAHVQOP365YZSW7BOFDTZYFFZ6TTKZBUVVIIWMN5AF6KTRDA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.562518953 (489h6m29.433601998s) [noise=true , address=145.79.227.51:4409 , piece_id=BRUMICPOT4MYZGRJV3FJLSFYR364TSV6SWAEFWT6LX5O5IOJBHAA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.743550842 (489h6m29.614633887s) [noise=true , address=145.79.227.2:4114 , piece_id=5XLLEUOHSCNSUGWRLWU4DRWMBCIFOQ5NDXY2TABK2YDCMQDQ2ZDQ, seg_pos={0 0}, upload=429285]: Done uploading piece. err: <nil> Dec 26 00:35:29.743569719 (489h6m29.614652754s) [seg_pos={0 0}, upload=429285]: Segment reached optimal threshold of 13 pieces. Dec 26 00:35:29.743735353 (489h6m29.614818388s) [noise=true , address=145.79.227.39:4013 , piece_id=BFUZMBVNECRRKWOB5JI7NV4E3UUDTLPRO3HIS643N74JET6PKBGA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: expected piece hash; context canceled; EOF Dec 26 00:35:29.743741475 (489h6m29.61482452s) [noise=true , address=145.79.227.39:4013 , piece_id=BFUZMBVNECRRKWOB5JI7NV4E3UUDTLPRO3HIS643N74JET6PKBGA, seg_pos={0 0}, upload=429285]: Long tail cancellation due to optimal threshold Dec 26 00:35:29.744010037 (489h6m29.615093072s) [noise=true , address=145.79.227.47:4265 , piece_id=L7PDIGOR3FA2V2YRH7VDZT4P3PO35P7MR7QEZ34PC3HYHEBC64IA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JvnJetuaEkstkPVNChKo8YQEYC68qRXPGQxx2AKZoHftkhajQ): protocol: write tcp 188.166.225.170:47712->145.79.227.47:4265: use of closed network connection; write tcp 188.166.225.170:47712->145.79.227.47:4265: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47712->145.79.227.47:4265: use of closed network connection Dec 26 00:35:29.744017120 (489h6m29.615100165s) [noise=true , address=145.79.227.47:4265 , piece_id=L7PDIGOR3FA2V2YRH7VDZT4P3PO35P7MR7QEZ34PC3HYHEBC64IA, seg_pos={0 0}, upload=429285]: Long tail cancellation due to optimal threshold Dec 26 00:35:29.744143180 (489h6m29.615226215s) [noise=true , address=145.79.227.15:4876 , piece_id=HQMR6OYIDWU2IHWAOFK3CTRVA5MGO4PMFWHYO536KCLLR3A5VMYA, seg_pos={0 0}, upload=429285]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: expected piece hash; context canceled; EOF Dec 26 00:35:29.744154011 (489h6m29.615237076s) [noise=true , address=145.79.227.15:4876 , piece_id=HQMR6OYIDWU2IHWAOFK3CTRVA5MGO4PMFWHYO536KCLLR3A5VMYA, seg_pos={0 0}, upload=429285]: Long tail cancellation due to optimal threshold Dec 26 00:35:29.744176834 (489h6m29.615259869s) [seg_pos={0 0}, upload=429285]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 00:35:29.744265102 (489h6m29.615348147s) [upload=429285]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 00:35:29.744271315 (489h6m29.61535436s) [upload=429285]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 00:35:30.052976317 (489h6m29.924059352s) [upload=429285]: Done uploading segments... Dec 26 00:35:30.052995132 (489h6m29.924078178s) [upload=429285]: Upload finished. err: <nil> Duration: 5.702226185s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)