Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..................X............................] duration: 31.449168312s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:26:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 31.449194672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............XXX..XXX......X..X.......X....X..............X..] duration: 3.650294077s trace: https://jaeger.dev.storj.io/trace/53df2664bdf722a6 FAIL (2025-12-29 10:26:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.650319665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.X..X..] duration: 5.994825152s trace: https://jaeger.dev.storj.io/trace/71e9635e0934e2b9 FAIL (2025-12-29 10:25:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q) context deadline exceeded ======LOGS====== ec 29 10:24:43.349957137 (570h55m43.221040172s) [upload=500539]: Starting upload Dec 29 10:24:43.350010078 (570h55m43.221093113s) [upload=500539]: Done starting upload Dec 29 10:24:43.350129325 (570h55m43.22121236s) [upload=500539]: Uploading segments... Dec 29 10:24:43.350220228 (570h55m43.221303263s) [upload=500539]: Got next segment. Inline: false Dec 29 10:24:43.350248712 (570h55m43.221331747s) [upload=500539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:24:43.350253802 (570h55m43.221336837s) [upload=500539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:24:43.595527721 (570h55m43.466610756s) [seg_pos={0 0}, upload=500539]: Begin upload segment... Dec 29 10:24:43.595571634 (570h55m43.466654679s) [seg_pos={0 0}, upload=500539]: Done begin upload segment. Dec 29 10:24:43.595598876 (570h55m43.466681921s) [noise=true , address=145.79.227.50:4375 , piece_id=USPB5BJGC43Q77REAMH3KSIC4U4G7BIWZT63DJKQ2NYFXD6TWWSQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.595672496 (570h55m43.466755541s) [noise=true , address=145.79.227.12:4765 , piece_id=PHZ2WWQVSVV2VKYXZO665MNE4MZ4CCOOZBEQKTBRFHXBIQKYKCRQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.596357651 (570h55m43.467440686s) [noise=true , address=145.79.227.14:4839 , piece_id=CADHXHESGRPU6F5ZC55QTWX3K5XI5Z6FCZPYEVBMWLFH7MS24JTQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.596667712 (570h55m43.467750777s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597035281 (570h55m43.468118326s) [noise=true , address=45.91.116.9:4264 , piece_id=MTLT2XQJY3VXDZKLXZETJCP5NHEDVPNQ46HDI4LSGCJXQFYXCL7A, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597412209 (570h55m43.468495284s) [noise=true , address=77.90.31.14:4664 , piece_id=IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597421506 (570h55m43.468504551s) [noise=true , address=145.79.227.48:4303 , piece_id=7MHR62XW4Y5ESEQ2PBUK44UUT6EVXYKQBQLLI33S5RP4XOIOHVXQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597724463 (570h55m43.468807509s) [noise=true , address=145.79.227.45:4231 , piece_id=GEUONEP7SWX4MWJTP47YBKDVXYEAF34D5ZWYVV6QPWI44DUH4EPA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598098195 (570h55m43.46918126s) [noise=true , address=145.79.227.37:4567 , piece_id=DEOSESSCMOF5L3FSMVWMOXK4MQGWQXDQLZVF35NO7GOK26XD6O7A, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598225518 (570h55m43.469308553s) [noise=true , address=77.90.31.15:4697 , piece_id=BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598524607 (570h55m43.469607652s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598602295 (570h55m43.46968533s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598822405 (570h55m43.46990544s) [noise=true , address=145.79.227.38:4979 , piece_id=PVMUIL3JGQO6RCULGFRQ6DVEY4PNE2QGWUMMOJB2TRK5PYO3BDHA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599155649 (570h55m43.470238694s) [noise=true , address=145.79.227.39:4015 , piece_id=BQ4EHXHSF7LNX6IE7V7GCSCIRLSTDNALCPZG23ZEWYIGOPXBX3EQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599522528 (570h55m43.470605563s) [noise=true , address=145.79.227.5:4223 , piece_id=Q3J257H5NW3TIHYBIK5BDKPQ44AC7ZCFXNVZQZBOFWAPWE7TVSAQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599963316 (570h55m43.471046361s) [noise=true , address=145.79.227.41:4085 , piece_id=PG5XCKRYRCLCLAK7KL6G3LJX2OOALAPPH44SBSZP6YB46FXQLIWQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:45.206614066 (570h55m45.077697101s) [upload=500539]: Next returned nil segment Dec 29 10:24:45.206625537 (570h55m45.077708572s) [upload=500539]: Waiting for error group managing segments... Dec 29 10:24:45.581682313 (570h55m45.452765358s) [noise=true , address=145.79.227.50:4375 , piece_id=USPB5BJGC43Q77REAMH3KSIC4U4G7BIWZT63DJKQ2NYFXD6TWWSQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.584757549 (570h55m45.455840594s) [noise=true , address=145.79.227.37:4567 , piece_id=DEOSESSCMOF5L3FSMVWMOXK4MQGWQXDQLZVF35NO7GOK26XD6O7A, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.587003446 (570h55m45.458086491s) [noise=true , address=145.79.227.38:4979 , piece_id=PVMUIL3JGQO6RCULGFRQ6DVEY4PNE2QGWUMMOJB2TRK5PYO3BDHA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.587155334 (570h55m45.458238369s) [noise=true , address=145.79.227.48:4303 , piece_id=7MHR62XW4Y5ESEQ2PBUK44UUT6EVXYKQBQLLI33S5RP4XOIOHVXQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.588459358 (570h55m45.459542393s) [noise=true , address=145.79.227.45:4231 , piece_id=GEUONEP7SWX4MWJTP47YBKDVXYEAF34D5ZWYVV6QPWI44DUH4EPA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.589259041 (570h55m45.460342086s) [noise=true , address=145.79.227.41:4085 , piece_id=PG5XCKRYRCLCLAK7KL6G3LJX2OOALAPPH44SBSZP6YB46FXQLIWQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.590522496 (570h55m45.461605531s) [noise=true , address=145.79.227.14:4839 , piece_id=CADHXHESGRPU6F5ZC55QTWX3K5XI5Z6FCZPYEVBMWLFH7MS24JTQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.605263956 (570h55m45.476347001s) [noise=true , address=145.79.227.39:4015 , piece_id=BQ4EHXHSF7LNX6IE7V7GCSCIRLSTDNALCPZG23ZEWYIGOPXBX3EQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.612023137 (570h55m45.483106182s) [noise=true , address=145.79.227.12:4765 , piece_id=PHZ2WWQVSVV2VKYXZO665MNE4MZ4CCOOZBEQKTBRFHXBIQKYKCRQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.779055973 (570h55m45.650139008s) [noise=true , address=77.90.31.15:4697 , piece_id=BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.943810771 (570h55m45.814893806s) [noise=true , address=45.91.116.9:4264 , piece_id=MTLT2XQJY3VXDZKLXZETJCP5NHEDVPNQ46HDI4LSGCJXQFYXCL7A, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.175635882 (570h55m46.046718917s) [noise=true , address=77.90.31.14:4664 , piece_id=IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.343313075 (570h55m46.21439611s) [noise=true , address=145.79.227.5:4223 , piece_id=Q3J257H5NW3TIHYBIK5BDKPQ44AC7ZCFXNVZQZBOFWAPWE7TVSAQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.343331300 (570h55m46.214414335s) [seg_pos={0 0}, upload=500539]: Segment reached optimal threshold of 13 pieces. Dec 29 10:24:46.343494711 (570h55m46.214577756s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection Dec 29 10:24:46.343501634 (570h55m46.214584669s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343581647 (570h55m46.214664682s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 10:24:46.343591175 (570h55m46.21467422s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343705783 (570h55m46.214788818s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:24:46.343712856 (570h55m46.214795891s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343729108 (570h55m46.214812143s) [seg_pos={0 0}, upload=500539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:24:46.343815562 (570h55m46.214898607s) [upload=500539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:24:46.343821323 (570h55m46.214904358s) [upload=500539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:24:46.639877921 (570h55m46.510960956s) [upload=500539]: Done uploading segments... Dec 29 10:24:46.639943215 (570h55m46.51102625s) [upload=500539]: Upload finished. err: <nil> Duration: 5.994841374s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [......XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.X..X.] duration: 27.15655586s trace: https://jaeger.dev.storj.io/trace/71e9635e0934e2b9 FAIL (2025-12-29 10:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q) context deadline exceeded ======LOGS====== ec 29 10:24:43.349957137 (570h55m43.221040172s) [upload=500539]: Starting upload Dec 29 10:24:43.350010078 (570h55m43.221093113s) [upload=500539]: Done starting upload Dec 29 10:24:43.350129325 (570h55m43.22121236s) [upload=500539]: Uploading segments... Dec 29 10:24:43.350220228 (570h55m43.221303263s) [upload=500539]: Got next segment. Inline: false Dec 29 10:24:43.350248712 (570h55m43.221331747s) [upload=500539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:24:43.350253802 (570h55m43.221336837s) [upload=500539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:24:43.595527721 (570h55m43.466610756s) [seg_pos={0 0}, upload=500539]: Begin upload segment... Dec 29 10:24:43.595571634 (570h55m43.466654679s) [seg_pos={0 0}, upload=500539]: Done begin upload segment. Dec 29 10:24:43.595598876 (570h55m43.466681921s) [noise=true , address=145.79.227.50:4375 , piece_id=USPB5BJGC43Q77REAMH3KSIC4U4G7BIWZT63DJKQ2NYFXD6TWWSQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.595672496 (570h55m43.466755541s) [noise=true , address=145.79.227.12:4765 , piece_id=PHZ2WWQVSVV2VKYXZO665MNE4MZ4CCOOZBEQKTBRFHXBIQKYKCRQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.596357651 (570h55m43.467440686s) [noise=true , address=145.79.227.14:4839 , piece_id=CADHXHESGRPU6F5ZC55QTWX3K5XI5Z6FCZPYEVBMWLFH7MS24JTQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.596667712 (570h55m43.467750777s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597035281 (570h55m43.468118326s) [noise=true , address=45.91.116.9:4264 , piece_id=MTLT2XQJY3VXDZKLXZETJCP5NHEDVPNQ46HDI4LSGCJXQFYXCL7A, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597412209 (570h55m43.468495284s) [noise=true , address=77.90.31.14:4664 , piece_id=IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597421506 (570h55m43.468504551s) [noise=true , address=145.79.227.48:4303 , piece_id=7MHR62XW4Y5ESEQ2PBUK44UUT6EVXYKQBQLLI33S5RP4XOIOHVXQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597724463 (570h55m43.468807509s) [noise=true , address=145.79.227.45:4231 , piece_id=GEUONEP7SWX4MWJTP47YBKDVXYEAF34D5ZWYVV6QPWI44DUH4EPA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598098195 (570h55m43.46918126s) [noise=true , address=145.79.227.37:4567 , piece_id=DEOSESSCMOF5L3FSMVWMOXK4MQGWQXDQLZVF35NO7GOK26XD6O7A, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598225518 (570h55m43.469308553s) [noise=true , address=77.90.31.15:4697 , piece_id=BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598524607 (570h55m43.469607652s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598602295 (570h55m43.46968533s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598822405 (570h55m43.46990544s) [noise=true , address=145.79.227.38:4979 , piece_id=PVMUIL3JGQO6RCULGFRQ6DVEY4PNE2QGWUMMOJB2TRK5PYO3BDHA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599155649 (570h55m43.470238694s) [noise=true , address=145.79.227.39:4015 , piece_id=BQ4EHXHSF7LNX6IE7V7GCSCIRLSTDNALCPZG23ZEWYIGOPXBX3EQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599522528 (570h55m43.470605563s) [noise=true , address=145.79.227.5:4223 , piece_id=Q3J257H5NW3TIHYBIK5BDKPQ44AC7ZCFXNVZQZBOFWAPWE7TVSAQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599963316 (570h55m43.471046361s) [noise=true , address=145.79.227.41:4085 , piece_id=PG5XCKRYRCLCLAK7KL6G3LJX2OOALAPPH44SBSZP6YB46FXQLIWQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:45.206614066 (570h55m45.077697101s) [upload=500539]: Next returned nil segment Dec 29 10:24:45.206625537 (570h55m45.077708572s) [upload=500539]: Waiting for error group managing segments... Dec 29 10:24:45.581682313 (570h55m45.452765358s) [noise=true , address=145.79.227.50:4375 , piece_id=USPB5BJGC43Q77REAMH3KSIC4U4G7BIWZT63DJKQ2NYFXD6TWWSQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.584757549 (570h55m45.455840594s) [noise=true , address=145.79.227.37:4567 , piece_id=DEOSESSCMOF5L3FSMVWMOXK4MQGWQXDQLZVF35NO7GOK26XD6O7A, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.587003446 (570h55m45.458086491s) [noise=true , address=145.79.227.38:4979 , piece_id=PVMUIL3JGQO6RCULGFRQ6DVEY4PNE2QGWUMMOJB2TRK5PYO3BDHA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.587155334 (570h55m45.458238369s) [noise=true , address=145.79.227.48:4303 , piece_id=7MHR62XW4Y5ESEQ2PBUK44UUT6EVXYKQBQLLI33S5RP4XOIOHVXQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.588459358 (570h55m45.459542393s) [noise=true , address=145.79.227.45:4231 , piece_id=GEUONEP7SWX4MWJTP47YBKDVXYEAF34D5ZWYVV6QPWI44DUH4EPA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.589259041 (570h55m45.460342086s) [noise=true , address=145.79.227.41:4085 , piece_id=PG5XCKRYRCLCLAK7KL6G3LJX2OOALAPPH44SBSZP6YB46FXQLIWQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.590522496 (570h55m45.461605531s) [noise=true , address=145.79.227.14:4839 , piece_id=CADHXHESGRPU6F5ZC55QTWX3K5XI5Z6FCZPYEVBMWLFH7MS24JTQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.605263956 (570h55m45.476347001s) [noise=true , address=145.79.227.39:4015 , piece_id=BQ4EHXHSF7LNX6IE7V7GCSCIRLSTDNALCPZG23ZEWYIGOPXBX3EQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.612023137 (570h55m45.483106182s) [noise=true , address=145.79.227.12:4765 , piece_id=PHZ2WWQVSVV2VKYXZO665MNE4MZ4CCOOZBEQKTBRFHXBIQKYKCRQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.779055973 (570h55m45.650139008s) [noise=true , address=77.90.31.15:4697 , piece_id=BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.943810771 (570h55m45.814893806s) [noise=true , address=45.91.116.9:4264 , piece_id=MTLT2XQJY3VXDZKLXZETJCP5NHEDVPNQ46HDI4LSGCJXQFYXCL7A, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.175635882 (570h55m46.046718917s) [noise=true , address=77.90.31.14:4664 , piece_id=IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.343313075 (570h55m46.21439611s) [noise=true , address=145.79.227.5:4223 , piece_id=Q3J257H5NW3TIHYBIK5BDKPQ44AC7ZCFXNVZQZBOFWAPWE7TVSAQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.343331300 (570h55m46.214414335s) [seg_pos={0 0}, upload=500539]: Segment reached optimal threshold of 13 pieces. Dec 29 10:24:46.343494711 (570h55m46.214577756s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection Dec 29 10:24:46.343501634 (570h55m46.214584669s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343581647 (570h55m46.214664682s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 10:24:46.343591175 (570h55m46.21467422s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343705783 (570h55m46.214788818s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:24:46.343712856 (570h55m46.214795891s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343729108 (570h55m46.214812143s) [seg_pos={0 0}, upload=500539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:24:46.343815562 (570h55m46.214898607s) [upload=500539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:24:46.343821323 (570h55m46.214904358s) [upload=500539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:24:46.639877921 (570h55m46.510960956s) [upload=500539]: Done uploading segments... Dec 29 10:24:46.639943215 (570h55m46.51102625s) [upload=500539]: Upload finished. err: <nil> Duration: 27.156584054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.............XXX..XXX......X..X.......X....X..............X.] duration: 3.440788196s trace: https://jaeger.dev.storj.io/trace/53df2664bdf722a6 FAIL (2025-12-29 10:25:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440813133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..................X...........................] duration: 15.388142625s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:25:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 15.388162593s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..................X..........................] duration: 35.892597429s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:25:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 35.892620493s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.......XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.X..X] duration: 30.002710753s trace: https://jaeger.dev.storj.io/trace/71e9635e0934e2b9 FAIL (2025-12-29 10:25:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q) context deadline exceeded ======LOGS====== ec 29 10:24:43.349957137 (570h55m43.221040172s) [upload=500539]: Starting upload Dec 29 10:24:43.350010078 (570h55m43.221093113s) [upload=500539]: Done starting upload Dec 29 10:24:43.350129325 (570h55m43.22121236s) [upload=500539]: Uploading segments... Dec 29 10:24:43.350220228 (570h55m43.221303263s) [upload=500539]: Got next segment. Inline: false Dec 29 10:24:43.350248712 (570h55m43.221331747s) [upload=500539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:24:43.350253802 (570h55m43.221336837s) [upload=500539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:24:43.595527721 (570h55m43.466610756s) [seg_pos={0 0}, upload=500539]: Begin upload segment... Dec 29 10:24:43.595571634 (570h55m43.466654679s) [seg_pos={0 0}, upload=500539]: Done begin upload segment. Dec 29 10:24:43.595598876 (570h55m43.466681921s) [noise=true , address=145.79.227.50:4375 , piece_id=USPB5BJGC43Q77REAMH3KSIC4U4G7BIWZT63DJKQ2NYFXD6TWWSQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.595672496 (570h55m43.466755541s) [noise=true , address=145.79.227.12:4765 , piece_id=PHZ2WWQVSVV2VKYXZO665MNE4MZ4CCOOZBEQKTBRFHXBIQKYKCRQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.596357651 (570h55m43.467440686s) [noise=true , address=145.79.227.14:4839 , piece_id=CADHXHESGRPU6F5ZC55QTWX3K5XI5Z6FCZPYEVBMWLFH7MS24JTQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.596667712 (570h55m43.467750777s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597035281 (570h55m43.468118326s) [noise=true , address=45.91.116.9:4264 , piece_id=MTLT2XQJY3VXDZKLXZETJCP5NHEDVPNQ46HDI4LSGCJXQFYXCL7A, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597412209 (570h55m43.468495284s) [noise=true , address=77.90.31.14:4664 , piece_id=IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597421506 (570h55m43.468504551s) [noise=true , address=145.79.227.48:4303 , piece_id=7MHR62XW4Y5ESEQ2PBUK44UUT6EVXYKQBQLLI33S5RP4XOIOHVXQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.597724463 (570h55m43.468807509s) [noise=true , address=145.79.227.45:4231 , piece_id=GEUONEP7SWX4MWJTP47YBKDVXYEAF34D5ZWYVV6QPWI44DUH4EPA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598098195 (570h55m43.46918126s) [noise=true , address=145.79.227.37:4567 , piece_id=DEOSESSCMOF5L3FSMVWMOXK4MQGWQXDQLZVF35NO7GOK26XD6O7A, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598225518 (570h55m43.469308553s) [noise=true , address=77.90.31.15:4697 , piece_id=BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598524607 (570h55m43.469607652s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598602295 (570h55m43.46968533s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.598822405 (570h55m43.46990544s) [noise=true , address=145.79.227.38:4979 , piece_id=PVMUIL3JGQO6RCULGFRQ6DVEY4PNE2QGWUMMOJB2TRK5PYO3BDHA, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599155649 (570h55m43.470238694s) [noise=true , address=145.79.227.39:4015 , piece_id=BQ4EHXHSF7LNX6IE7V7GCSCIRLSTDNALCPZG23ZEWYIGOPXBX3EQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599522528 (570h55m43.470605563s) [noise=true , address=145.79.227.5:4223 , piece_id=Q3J257H5NW3TIHYBIK5BDKPQ44AC7ZCFXNVZQZBOFWAPWE7TVSAQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:43.599963316 (570h55m43.471046361s) [noise=true , address=145.79.227.41:4085 , piece_id=PG5XCKRYRCLCLAK7KL6G3LJX2OOALAPPH44SBSZP6YB46FXQLIWQ, seg_pos={0 0}, upload=500539]: Uploading piece... Dec 29 10:24:45.206614066 (570h55m45.077697101s) [upload=500539]: Next returned nil segment Dec 29 10:24:45.206625537 (570h55m45.077708572s) [upload=500539]: Waiting for error group managing segments... Dec 29 10:24:45.581682313 (570h55m45.452765358s) [noise=true , address=145.79.227.50:4375 , piece_id=USPB5BJGC43Q77REAMH3KSIC4U4G7BIWZT63DJKQ2NYFXD6TWWSQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.584757549 (570h55m45.455840594s) [noise=true , address=145.79.227.37:4567 , piece_id=DEOSESSCMOF5L3FSMVWMOXK4MQGWQXDQLZVF35NO7GOK26XD6O7A, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.587003446 (570h55m45.458086491s) [noise=true , address=145.79.227.38:4979 , piece_id=PVMUIL3JGQO6RCULGFRQ6DVEY4PNE2QGWUMMOJB2TRK5PYO3BDHA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.587155334 (570h55m45.458238369s) [noise=true , address=145.79.227.48:4303 , piece_id=7MHR62XW4Y5ESEQ2PBUK44UUT6EVXYKQBQLLI33S5RP4XOIOHVXQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.588459358 (570h55m45.459542393s) [noise=true , address=145.79.227.45:4231 , piece_id=GEUONEP7SWX4MWJTP47YBKDVXYEAF34D5ZWYVV6QPWI44DUH4EPA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.589259041 (570h55m45.460342086s) [noise=true , address=145.79.227.41:4085 , piece_id=PG5XCKRYRCLCLAK7KL6G3LJX2OOALAPPH44SBSZP6YB46FXQLIWQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.590522496 (570h55m45.461605531s) [noise=true , address=145.79.227.14:4839 , piece_id=CADHXHESGRPU6F5ZC55QTWX3K5XI5Z6FCZPYEVBMWLFH7MS24JTQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.605263956 (570h55m45.476347001s) [noise=true , address=145.79.227.39:4015 , piece_id=BQ4EHXHSF7LNX6IE7V7GCSCIRLSTDNALCPZG23ZEWYIGOPXBX3EQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.612023137 (570h55m45.483106182s) [noise=true , address=145.79.227.12:4765 , piece_id=PHZ2WWQVSVV2VKYXZO665MNE4MZ4CCOOZBEQKTBRFHXBIQKYKCRQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.779055973 (570h55m45.650139008s) [noise=true , address=77.90.31.15:4697 , piece_id=BR3TLMA6XA27A644N5UR65MQBQBOYAJPH3IOGLMMZANVBEEZQV4Q, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:45.943810771 (570h55m45.814893806s) [noise=true , address=45.91.116.9:4264 , piece_id=MTLT2XQJY3VXDZKLXZETJCP5NHEDVPNQ46HDI4LSGCJXQFYXCL7A, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.175635882 (570h55m46.046718917s) [noise=true , address=77.90.31.14:4664 , piece_id=IZZ2HBCYCHJ5XUBRCNJ7M6BUBZG67KGM2E7VCLQKW725RLYIT2LQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.343313075 (570h55m46.21439611s) [noise=true , address=145.79.227.5:4223 , piece_id=Q3J257H5NW3TIHYBIK5BDKPQ44AC7ZCFXNVZQZBOFWAPWE7TVSAQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: <nil> Dec 29 10:24:46.343331300 (570h55m46.214414335s) [seg_pos={0 0}, upload=500539]: Segment reached optimal threshold of 13 pieces. Dec 29 10:24:46.343494711 (570h55m46.214577756s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56246->45.91.116.6:4975: use of closed network connection Dec 29 10:24:46.343501634 (570h55m46.214584669s) [noise=true , address=45.91.116.6:4975 , piece_id=D2LXEBFOMXGO6VJ64NORX4R47KTEMDFZKBWJHOKDVPGGBMBHNMCA, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343581647 (570h55m46.214664682s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 29 10:24:46.343591175 (570h55m46.21467422s) [noise=true , address=45.91.116.8:4046 , piece_id=ETZ7EOQ4NXTDEKVSUAEXZOELJWNQ3UWOBQME4WTB5DSU2GRWL5GA, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343705783 (570h55m46.214788818s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 10:24:46.343712856 (570h55m46.214795891s) [noise=true , address=145.79.227.43:4158 , piece_id=SQS2PENAISDNKHP3S6KZGOUMVXFVXPW7RW6ISI24L2XQD3RYF3TQ, seg_pos={0 0}, upload=500539]: Long tail cancellation due to optimal threshold Dec 29 10:24:46.343729108 (570h55m46.214812143s) [seg_pos={0 0}, upload=500539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:24:46.343815562 (570h55m46.214898607s) [upload=500539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:24:46.343821323 (570h55m46.214904358s) [upload=500539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:24:46.639877921 (570h55m46.510960956s) [upload=500539]: Done uploading segments... Dec 29 10:24:46.639943215 (570h55m46.51102625s) [upload=500539]: Upload finished. err: <nil> Duration: 30.002732063s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..............XXX..XXX......X..X.......X....X..............X] duration: 5.001205181s trace: https://jaeger.dev.storj.io/trace/53df2664bdf722a6 FAIL (2025-12-29 10:24:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001227604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [........XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.X..] duration: 5.704145547s trace: https://jaeger.dev.storj.io/trace/10ed92ec58bef9a5 FAIL (2025-12-29 10:24:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA) context deadline exceeded ======LOGS====== Dec 29 10:23:00.867570534 (570h54m0.738653579s) [upload=500513]: Starting upload Dec 29 10:23:00.867675374 (570h54m0.738758439s) [upload=500513]: Done starting upload Dec 29 10:23:00.867784211 (570h54m0.738867246s) [upload=500513]: Uploading segments... Dec 29 10:23:00.867955226 (570h54m0.739038262s) [upload=500513]: Got next segment. Inline: false Dec 29 10:23:00.868005954 (570h54m0.739088989s) [upload=500513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:23:00.868011664 (570h54m0.739094699s) [upload=500513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:23:01.119188333 (570h54m0.990271378s) [seg_pos={0 0}, upload=500513]: Begin upload segment... Dec 29 10:23:01.119262343 (570h54m0.990345388s) [seg_pos={0 0}, upload=500513]: Done begin upload segment. Dec 29 10:23:01.119359789 (570h54m0.990442824s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.119380850 (570h54m0.990463885s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120033132 (570h54m0.991116167s) [noise=true , address=77.90.31.14:4664 , piece_id=ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120575865 (570h54m0.9916589s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120606042 (570h54m0.991689087s) [noise=true , address=145.79.227.28:4725 , piece_id=NI5W5PTHM6IF6IQ67UPRXFXYRJJYOUT3LNTCTVS4QBC6QBDTM6PQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120914710 (570h54m0.991997745s) [noise=true , address=145.79.227.19:4077 , piece_id=CDWXIM2N7MOVISCTXNLSWUAPSWNC5OREDS373EPRPER3CRDLPX7Q, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121036301 (570h54m0.992119336s) [noise=true , address=145.79.227.46:4601 , piece_id=ZOAHPQFQEHRVHJVY2L4VON722HHFZWPT4SBHFOC55MGHZXI2ZTBQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121253416 (570h54m0.992336461s) [noise=true , address=145.79.227.15:4874 , piece_id=7YLJRKDR3BNPFWWIMDGVDUIFGRRUJNSVTQVQFS3VC4WTJQZHNUHA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121452103 (570h54m0.992535148s) [noise=true , address=45.91.116.10:4084 , piece_id=F5WG2HAAYSEKHDN6BHSBD36SG3YO4HFBIQGXRL6EUFWDFGH3UGHA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121628509 (570h54m0.992711544s) [noise=true , address=45.91.116.7:4012 , piece_id=N54LVHS7RM2VTEHWAABOJYZCBHKD4MBQVDJKM2V3LW2YHUDBA7SQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122109365 (570h54m0.9931924s) [noise=true , address=77.90.31.16:4735 , piece_id=NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122222411 (570h54m0.993305446s) [noise=true , address=145.79.227.52:4446 , piece_id=CAGQMUZOCGAXTUWLXJB2RG3JULM56XNCA3ASYC3TRWOWZL2NZNWQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122577817 (570h54m0.993660852s) [noise=true , address=145.79.227.3:4150 , piece_id=ANRQ2MTTM7RT7IILQZNFZXVUMY36JZKPJPNUTTQNNZTGFXTW6KBQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.123494152 (570h54m0.994577187s) [noise=true , address=145.79.227.38:4978 , piece_id=FBS24QLSEI2HN3K6YUTMFACS2YA4FEWU7V6AJ423WSDNCQCZTG2Q, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.124278847 (570h54m0.995361882s) [noise=true , address=145.79.227.16:4909 , piece_id=RFQBUWAEKZRJTI2HLYM3APK2TV2NT24FDM5P2DZFBJLK3ZFOELHQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.125194560 (570h54m0.996277595s) [noise=true , address=145.79.227.4:4185 , piece_id=S2EBWW52CYMTZLX6AWSWDNKU2TXKQ3WKYBWYT5D5ZRVSLCQI6BFA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:02.825842225 (570h54m2.69692526s) [upload=500513]: Next returned nil segment Dec 29 10:23:02.825857704 (570h54m2.696940749s) [upload=500513]: Waiting for error group managing segments... Dec 29 10:23:03.114448655 (570h54m2.9855317s) [noise=true , address=145.79.227.38:4978 , piece_id=FBS24QLSEI2HN3K6YUTMFACS2YA4FEWU7V6AJ423WSDNCQCZTG2Q, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.114577341 (570h54m2.985660386s) [noise=true , address=145.79.227.19:4077 , piece_id=CDWXIM2N7MOVISCTXNLSWUAPSWNC5OREDS373EPRPER3CRDLPX7Q, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.117409834 (570h54m2.988492879s) [noise=true , address=145.79.227.4:4185 , piece_id=S2EBWW52CYMTZLX6AWSWDNKU2TXKQ3WKYBWYT5D5ZRVSLCQI6BFA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.120260713 (570h54m2.991343768s) [noise=true , address=145.79.227.46:4601 , piece_id=ZOAHPQFQEHRVHJVY2L4VON722HHFZWPT4SBHFOC55MGHZXI2ZTBQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.121180994 (570h54m2.99226404s) [noise=true , address=145.79.227.28:4725 , piece_id=NI5W5PTHM6IF6IQ67UPRXFXYRJJYOUT3LNTCTVS4QBC6QBDTM6PQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.122394976 (570h54m2.993478021s) [noise=true , address=145.79.227.3:4150 , piece_id=ANRQ2MTTM7RT7IILQZNFZXVUMY36JZKPJPNUTTQNNZTGFXTW6KBQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.123811103 (570h54m2.994894138s) [noise=true , address=145.79.227.52:4446 , piece_id=CAGQMUZOCGAXTUWLXJB2RG3JULM56XNCA3ASYC3TRWOWZL2NZNWQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.135298447 (570h54m3.006381492s) [noise=true , address=145.79.227.15:4874 , piece_id=7YLJRKDR3BNPFWWIMDGVDUIFGRRUJNSVTQVQFS3VC4WTJQZHNUHA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.140519689 (570h54m3.011602734s) [noise=true , address=145.79.227.16:4909 , piece_id=RFQBUWAEKZRJTI2HLYM3APK2TV2NT24FDM5P2DZFBJLK3ZFOELHQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.387493054 (570h54m3.258576099s) [noise=true , address=45.91.116.7:4012 , piece_id=N54LVHS7RM2VTEHWAABOJYZCBHKD4MBQVDJKM2V3LW2YHUDBA7SQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.498909818 (570h54m3.369992864s) [noise=true , address=45.91.116.10:4084 , piece_id=F5WG2HAAYSEKHDN6BHSBD36SG3YO4HFBIQGXRL6EUFWDFGH3UGHA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.721293362 (570h54m3.592376397s) [noise=true , address=77.90.31.16:4735 , piece_id=NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.824989425 (570h54m3.69607247s) [noise=true , address=77.90.31.14:4664 , piece_id=ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.825017629 (570h54m3.696100664s) [seg_pos={0 0}, upload=500513]: Segment reached optimal threshold of 13 pieces. Dec 29 10:23:03.825269448 (570h54m3.696352503s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 10:23:03.825278946 (570h54m3.696361991s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825458398 (570h54m3.696541453s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection Dec 29 10:23:03.825531898 (570h54m3.696614943s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825746035 (570h54m3.69682911s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 10:23:03.825758159 (570h54m3.696841204s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825834895 (570h54m3.69691794s) [seg_pos={0 0}, upload=500513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:23:03.826296313 (570h54m3.697379358s) [upload=500513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:23:03.826306022 (570h54m3.697389057s) [upload=500513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:23:04.126406178 (570h54m3.997489213s) [upload=500513]: Done uploading segments... Dec 29 10:23:04.126432909 (570h54m3.997515954s) [upload=500513]: Upload finished. err: <nil> Duration: 5.704175495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.X.] duration: 6.096266687s trace: https://jaeger.dev.storj.io/trace/10ed92ec58bef9a5 FAIL (2025-12-29 10:23:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA) context deadline exceeded ======LOGS====== Dec 29 10:23:00.867570534 (570h54m0.738653579s) [upload=500513]: Starting upload Dec 29 10:23:00.867675374 (570h54m0.738758439s) [upload=500513]: Done starting upload Dec 29 10:23:00.867784211 (570h54m0.738867246s) [upload=500513]: Uploading segments... Dec 29 10:23:00.867955226 (570h54m0.739038262s) [upload=500513]: Got next segment. Inline: false Dec 29 10:23:00.868005954 (570h54m0.739088989s) [upload=500513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:23:00.868011664 (570h54m0.739094699s) [upload=500513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:23:01.119188333 (570h54m0.990271378s) [seg_pos={0 0}, upload=500513]: Begin upload segment... Dec 29 10:23:01.119262343 (570h54m0.990345388s) [seg_pos={0 0}, upload=500513]: Done begin upload segment. Dec 29 10:23:01.119359789 (570h54m0.990442824s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.119380850 (570h54m0.990463885s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120033132 (570h54m0.991116167s) [noise=true , address=77.90.31.14:4664 , piece_id=ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120575865 (570h54m0.9916589s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120606042 (570h54m0.991689087s) [noise=true , address=145.79.227.28:4725 , piece_id=NI5W5PTHM6IF6IQ67UPRXFXYRJJYOUT3LNTCTVS4QBC6QBDTM6PQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120914710 (570h54m0.991997745s) [noise=true , address=145.79.227.19:4077 , piece_id=CDWXIM2N7MOVISCTXNLSWUAPSWNC5OREDS373EPRPER3CRDLPX7Q, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121036301 (570h54m0.992119336s) [noise=true , address=145.79.227.46:4601 , piece_id=ZOAHPQFQEHRVHJVY2L4VON722HHFZWPT4SBHFOC55MGHZXI2ZTBQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121253416 (570h54m0.992336461s) [noise=true , address=145.79.227.15:4874 , piece_id=7YLJRKDR3BNPFWWIMDGVDUIFGRRUJNSVTQVQFS3VC4WTJQZHNUHA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121452103 (570h54m0.992535148s) [noise=true , address=45.91.116.10:4084 , piece_id=F5WG2HAAYSEKHDN6BHSBD36SG3YO4HFBIQGXRL6EUFWDFGH3UGHA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121628509 (570h54m0.992711544s) [noise=true , address=45.91.116.7:4012 , piece_id=N54LVHS7RM2VTEHWAABOJYZCBHKD4MBQVDJKM2V3LW2YHUDBA7SQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122109365 (570h54m0.9931924s) [noise=true , address=77.90.31.16:4735 , piece_id=NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122222411 (570h54m0.993305446s) [noise=true , address=145.79.227.52:4446 , piece_id=CAGQMUZOCGAXTUWLXJB2RG3JULM56XNCA3ASYC3TRWOWZL2NZNWQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122577817 (570h54m0.993660852s) [noise=true , address=145.79.227.3:4150 , piece_id=ANRQ2MTTM7RT7IILQZNFZXVUMY36JZKPJPNUTTQNNZTGFXTW6KBQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.123494152 (570h54m0.994577187s) [noise=true , address=145.79.227.38:4978 , piece_id=FBS24QLSEI2HN3K6YUTMFACS2YA4FEWU7V6AJ423WSDNCQCZTG2Q, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.124278847 (570h54m0.995361882s) [noise=true , address=145.79.227.16:4909 , piece_id=RFQBUWAEKZRJTI2HLYM3APK2TV2NT24FDM5P2DZFBJLK3ZFOELHQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.125194560 (570h54m0.996277595s) [noise=true , address=145.79.227.4:4185 , piece_id=S2EBWW52CYMTZLX6AWSWDNKU2TXKQ3WKYBWYT5D5ZRVSLCQI6BFA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:02.825842225 (570h54m2.69692526s) [upload=500513]: Next returned nil segment Dec 29 10:23:02.825857704 (570h54m2.696940749s) [upload=500513]: Waiting for error group managing segments... Dec 29 10:23:03.114448655 (570h54m2.9855317s) [noise=true , address=145.79.227.38:4978 , piece_id=FBS24QLSEI2HN3K6YUTMFACS2YA4FEWU7V6AJ423WSDNCQCZTG2Q, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.114577341 (570h54m2.985660386s) [noise=true , address=145.79.227.19:4077 , piece_id=CDWXIM2N7MOVISCTXNLSWUAPSWNC5OREDS373EPRPER3CRDLPX7Q, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.117409834 (570h54m2.988492879s) [noise=true , address=145.79.227.4:4185 , piece_id=S2EBWW52CYMTZLX6AWSWDNKU2TXKQ3WKYBWYT5D5ZRVSLCQI6BFA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.120260713 (570h54m2.991343768s) [noise=true , address=145.79.227.46:4601 , piece_id=ZOAHPQFQEHRVHJVY2L4VON722HHFZWPT4SBHFOC55MGHZXI2ZTBQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.121180994 (570h54m2.99226404s) [noise=true , address=145.79.227.28:4725 , piece_id=NI5W5PTHM6IF6IQ67UPRXFXYRJJYOUT3LNTCTVS4QBC6QBDTM6PQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.122394976 (570h54m2.993478021s) [noise=true , address=145.79.227.3:4150 , piece_id=ANRQ2MTTM7RT7IILQZNFZXVUMY36JZKPJPNUTTQNNZTGFXTW6KBQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.123811103 (570h54m2.994894138s) [noise=true , address=145.79.227.52:4446 , piece_id=CAGQMUZOCGAXTUWLXJB2RG3JULM56XNCA3ASYC3TRWOWZL2NZNWQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.135298447 (570h54m3.006381492s) [noise=true , address=145.79.227.15:4874 , piece_id=7YLJRKDR3BNPFWWIMDGVDUIFGRRUJNSVTQVQFS3VC4WTJQZHNUHA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.140519689 (570h54m3.011602734s) [noise=true , address=145.79.227.16:4909 , piece_id=RFQBUWAEKZRJTI2HLYM3APK2TV2NT24FDM5P2DZFBJLK3ZFOELHQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.387493054 (570h54m3.258576099s) [noise=true , address=45.91.116.7:4012 , piece_id=N54LVHS7RM2VTEHWAABOJYZCBHKD4MBQVDJKM2V3LW2YHUDBA7SQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.498909818 (570h54m3.369992864s) [noise=true , address=45.91.116.10:4084 , piece_id=F5WG2HAAYSEKHDN6BHSBD36SG3YO4HFBIQGXRL6EUFWDFGH3UGHA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.721293362 (570h54m3.592376397s) [noise=true , address=77.90.31.16:4735 , piece_id=NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.824989425 (570h54m3.69607247s) [noise=true , address=77.90.31.14:4664 , piece_id=ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.825017629 (570h54m3.696100664s) [seg_pos={0 0}, upload=500513]: Segment reached optimal threshold of 13 pieces. Dec 29 10:23:03.825269448 (570h54m3.696352503s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 10:23:03.825278946 (570h54m3.696361991s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825458398 (570h54m3.696541453s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection Dec 29 10:23:03.825531898 (570h54m3.696614943s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825746035 (570h54m3.69682911s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 10:23:03.825758159 (570h54m3.696841204s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825834895 (570h54m3.69691794s) [seg_pos={0 0}, upload=500513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:23:03.826296313 (570h54m3.697379358s) [upload=500513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:23:03.826306022 (570h54m3.697389057s) [upload=500513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:23:04.126406178 (570h54m3.997489213s) [upload=500513]: Done uploading segments... Dec 29 10:23:04.126432909 (570h54m3.997515954s) [upload=500513]: Upload finished. err: <nil> Duration: 6.096295381s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..................X.........................] duration: 17.930092897s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:23:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 17.930109748s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...............XXX..XXX......X..X.......X....X..............] duration: 3.484813282s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:23:36 UTC) orbiter warning: context deadline exceeded Duration: 3.484838108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..................X........................] duration: 48.724603367s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:23:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 48.724626891s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..........XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.X] duration: 30.003106821s trace: https://jaeger.dev.storj.io/trace/10ed92ec58bef9a5 FAIL (2025-12-29 10:23:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA) context deadline exceeded ======LOGS====== Dec 29 10:23:00.867570534 (570h54m0.738653579s) [upload=500513]: Starting upload Dec 29 10:23:00.867675374 (570h54m0.738758439s) [upload=500513]: Done starting upload Dec 29 10:23:00.867784211 (570h54m0.738867246s) [upload=500513]: Uploading segments... Dec 29 10:23:00.867955226 (570h54m0.739038262s) [upload=500513]: Got next segment. Inline: false Dec 29 10:23:00.868005954 (570h54m0.739088989s) [upload=500513]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:23:00.868011664 (570h54m0.739094699s) [upload=500513]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:23:01.119188333 (570h54m0.990271378s) [seg_pos={0 0}, upload=500513]: Begin upload segment... Dec 29 10:23:01.119262343 (570h54m0.990345388s) [seg_pos={0 0}, upload=500513]: Done begin upload segment. Dec 29 10:23:01.119359789 (570h54m0.990442824s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.119380850 (570h54m0.990463885s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120033132 (570h54m0.991116167s) [noise=true , address=77.90.31.14:4664 , piece_id=ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120575865 (570h54m0.9916589s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120606042 (570h54m0.991689087s) [noise=true , address=145.79.227.28:4725 , piece_id=NI5W5PTHM6IF6IQ67UPRXFXYRJJYOUT3LNTCTVS4QBC6QBDTM6PQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.120914710 (570h54m0.991997745s) [noise=true , address=145.79.227.19:4077 , piece_id=CDWXIM2N7MOVISCTXNLSWUAPSWNC5OREDS373EPRPER3CRDLPX7Q, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121036301 (570h54m0.992119336s) [noise=true , address=145.79.227.46:4601 , piece_id=ZOAHPQFQEHRVHJVY2L4VON722HHFZWPT4SBHFOC55MGHZXI2ZTBQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121253416 (570h54m0.992336461s) [noise=true , address=145.79.227.15:4874 , piece_id=7YLJRKDR3BNPFWWIMDGVDUIFGRRUJNSVTQVQFS3VC4WTJQZHNUHA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121452103 (570h54m0.992535148s) [noise=true , address=45.91.116.10:4084 , piece_id=F5WG2HAAYSEKHDN6BHSBD36SG3YO4HFBIQGXRL6EUFWDFGH3UGHA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.121628509 (570h54m0.992711544s) [noise=true , address=45.91.116.7:4012 , piece_id=N54LVHS7RM2VTEHWAABOJYZCBHKD4MBQVDJKM2V3LW2YHUDBA7SQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122109365 (570h54m0.9931924s) [noise=true , address=77.90.31.16:4735 , piece_id=NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122222411 (570h54m0.993305446s) [noise=true , address=145.79.227.52:4446 , piece_id=CAGQMUZOCGAXTUWLXJB2RG3JULM56XNCA3ASYC3TRWOWZL2NZNWQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.122577817 (570h54m0.993660852s) [noise=true , address=145.79.227.3:4150 , piece_id=ANRQ2MTTM7RT7IILQZNFZXVUMY36JZKPJPNUTTQNNZTGFXTW6KBQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.123494152 (570h54m0.994577187s) [noise=true , address=145.79.227.38:4978 , piece_id=FBS24QLSEI2HN3K6YUTMFACS2YA4FEWU7V6AJ423WSDNCQCZTG2Q, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.124278847 (570h54m0.995361882s) [noise=true , address=145.79.227.16:4909 , piece_id=RFQBUWAEKZRJTI2HLYM3APK2TV2NT24FDM5P2DZFBJLK3ZFOELHQ, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:01.125194560 (570h54m0.996277595s) [noise=true , address=145.79.227.4:4185 , piece_id=S2EBWW52CYMTZLX6AWSWDNKU2TXKQ3WKYBWYT5D5ZRVSLCQI6BFA, seg_pos={0 0}, upload=500513]: Uploading piece... Dec 29 10:23:02.825842225 (570h54m2.69692526s) [upload=500513]: Next returned nil segment Dec 29 10:23:02.825857704 (570h54m2.696940749s) [upload=500513]: Waiting for error group managing segments... Dec 29 10:23:03.114448655 (570h54m2.9855317s) [noise=true , address=145.79.227.38:4978 , piece_id=FBS24QLSEI2HN3K6YUTMFACS2YA4FEWU7V6AJ423WSDNCQCZTG2Q, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.114577341 (570h54m2.985660386s) [noise=true , address=145.79.227.19:4077 , piece_id=CDWXIM2N7MOVISCTXNLSWUAPSWNC5OREDS373EPRPER3CRDLPX7Q, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.117409834 (570h54m2.988492879s) [noise=true , address=145.79.227.4:4185 , piece_id=S2EBWW52CYMTZLX6AWSWDNKU2TXKQ3WKYBWYT5D5ZRVSLCQI6BFA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.120260713 (570h54m2.991343768s) [noise=true , address=145.79.227.46:4601 , piece_id=ZOAHPQFQEHRVHJVY2L4VON722HHFZWPT4SBHFOC55MGHZXI2ZTBQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.121180994 (570h54m2.99226404s) [noise=true , address=145.79.227.28:4725 , piece_id=NI5W5PTHM6IF6IQ67UPRXFXYRJJYOUT3LNTCTVS4QBC6QBDTM6PQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.122394976 (570h54m2.993478021s) [noise=true , address=145.79.227.3:4150 , piece_id=ANRQ2MTTM7RT7IILQZNFZXVUMY36JZKPJPNUTTQNNZTGFXTW6KBQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.123811103 (570h54m2.994894138s) [noise=true , address=145.79.227.52:4446 , piece_id=CAGQMUZOCGAXTUWLXJB2RG3JULM56XNCA3ASYC3TRWOWZL2NZNWQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.135298447 (570h54m3.006381492s) [noise=true , address=145.79.227.15:4874 , piece_id=7YLJRKDR3BNPFWWIMDGVDUIFGRRUJNSVTQVQFS3VC4WTJQZHNUHA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.140519689 (570h54m3.011602734s) [noise=true , address=145.79.227.16:4909 , piece_id=RFQBUWAEKZRJTI2HLYM3APK2TV2NT24FDM5P2DZFBJLK3ZFOELHQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.387493054 (570h54m3.258576099s) [noise=true , address=45.91.116.7:4012 , piece_id=N54LVHS7RM2VTEHWAABOJYZCBHKD4MBQVDJKM2V3LW2YHUDBA7SQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.498909818 (570h54m3.369992864s) [noise=true , address=45.91.116.10:4084 , piece_id=F5WG2HAAYSEKHDN6BHSBD36SG3YO4HFBIQGXRL6EUFWDFGH3UGHA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.721293362 (570h54m3.592376397s) [noise=true , address=77.90.31.16:4735 , piece_id=NBCMOBPW2SIQWHZEHZM3MQPCVEJL6BJXRGOURZQ3DFZJSEUWA3OA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.824989425 (570h54m3.69607247s) [noise=true , address=77.90.31.14:4664 , piece_id=ZWSQQXDKSI7INWGKGOBWQCOSLPD7C332AXCH3EVV5C646YQLZ5HQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: <nil> Dec 29 10:23:03.825017629 (570h54m3.696100664s) [seg_pos={0 0}, upload=500513]: Segment reached optimal threshold of 13 pieces. Dec 29 10:23:03.825269448 (570h54m3.696352503s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 29 10:23:03.825278946 (570h54m3.696361991s) [noise=true , address=45.91.116.11:4118 , piece_id=F2Q6XSD7POZAGV6J7RJSFAVAASN6QTCKHVUXGYMS2LODISMC2B5A, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825458398 (570h54m3.696541453s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49950->45.91.116.13:4156: use of closed network connection Dec 29 10:23:03.825531898 (570h54m3.696614943s) [noise=true , address=45.91.116.13:4156 , piece_id=4ALAEDIIDTSEZKTYGPR2DNLBUFVQ7EIJT6U6PUK7HY6KAKQY5POQ, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825746035 (570h54m3.69682911s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 29 10:23:03.825758159 (570h54m3.696841204s) [noise=true , address=77.90.31.17:4770 , piece_id=EFHISBAFVDZT3H6JKSKI2QV5MTSHFSJIPG2GX7QAPQUGIKVFU4WA, seg_pos={0 0}, upload=500513]: Long tail cancellation due to optimal threshold Dec 29 10:23:03.825834895 (570h54m3.69691794s) [seg_pos={0 0}, upload=500513]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:23:03.826296313 (570h54m3.697379358s) [upload=500513]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:23:03.826306022 (570h54m3.697389057s) [upload=500513]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:23:04.126406178 (570h54m3.997489213s) [upload=500513]: Done uploading segments... Dec 29 10:23:04.126432909 (570h54m3.997515954s) [upload=500513]: Upload finished. err: <nil> Duration: 30.003142889s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [................XXX..XXX......X..X.......X....X.............] duration: 5.00075832s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:22:53 UTC) orbiter warning: context deadline exceeded Duration: 5.000779932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..................X.......................] duration: 40.242371073s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:22:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 40.242384961s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..........XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X.] duration: 5.946734402s trace: https://jaeger.dev.storj.io/trace/71c8c8bce954bac8 FAIL (2025-12-29 10:22:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XKQK4X5YYM4MZI7KA7HDHZHB5IVGDAKAE4IT6SOZRUU4YQB3WYLA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: KCLF74EIYNTZ2QIWN2YDEJLNXJYMKHZYFDF62DLFTZMS54PI3LFQ) context deadline exceeded ======LOGS====== Dec 29 10:20:57.804358690 (570h51m57.675441725s) [upload=500484]: Starting upload Dec 29 10:20:57.804417613 (570h51m57.675500648s) [upload=500484]: Done starting upload Dec 29 10:20:57.804589891 (570h51m57.675672926s) [upload=500484]: Uploading segments... Dec 29 10:20:57.804623735 (570h51m57.675706771s) [upload=500484]: Got next segment. Inline: false Dec 29 10:20:57.804647822 (570h51m57.675730857s) [upload=500484]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:20:57.804651829 (570h51m57.675734864s) [upload=500484]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:20:58.049104354 (570h51m57.920187389s) [seg_pos={0 0}, upload=500484]: Begin upload segment... Dec 29 10:20:58.049143959 (570h51m57.920227004s) [seg_pos={0 0}, upload=500484]: Done begin upload segment. Dec 29 10:20:58.049169117 (570h51m57.920252152s) [noise=true , address=45.91.116.8:4047 , piece_id=XKQK4X5YYM4MZI7KA7HDHZHB5IVGDAKAE4IT6SOZRUU4YQB3WYLA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.050248312 (570h51m57.921331377s) [noise=true , address=145.79.227.51:4410 , piece_id=BACJKUWO6EF647ZQTEHBC4LBDEUCL5K5VLVQ4Q7RXEVGITK5YC7Q, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.050706204 (570h51m57.921789239s) [noise=true , address=145.79.227.37:4566 , piece_id=YI4CN6NW7F2QEIKWZG7QOBQVKLYR7K7DX764RH63KLA5W4JQIJVQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.051111516 (570h51m57.922194551s) [noise=true , address=145.79.227.2:4114 , piece_id=COQAGJFIQIDWPL245JNAJJAC4QPS37SNDKBDONAHJX77T4IIB5RQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.051569468 (570h51m57.922652503s) [noise=true , address=45.91.116.11:4119 , piece_id=WGNYWLKUJV4BNSJMNFAX3G74KY4UCYEB76QCBORTIGTCQN4DXXKA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.051969198 (570h51m57.923052234s) [noise=true , address=145.79.227.13:4804 , piece_id=FIEA2XHVDBSDA3W7XKQOJGTFODF7GONPTO3E5LOEP4CVERSMCROQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.052297695 (570h51m57.92338073s) [noise=true , address=145.79.227.47:4265 , piece_id=LKTFJYA4EJWWNH35FXAJRN7XLK2Y5L7UMCCRAURKRU7DSG4E5A4A, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.052631500 (570h51m57.923714535s) [noise=true , address=45.91.116.14:4191 , piece_id=KCLF74EIYNTZ2QIWN2YDEJLNXJYMKHZYFDF62DLFTZMS54PI3LFQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.053065376 (570h51m57.924148411s) [noise=true , address=145.79.227.6:4258 , piece_id=CUNQGOJXPUGWRWHA5TDSXMYFTEZ7KSTMAJQ6RBJFFRK5VUYSPNBA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.053458164 (570h51m57.924541199s) [noise=true , address=145.79.227.40:4050 , piece_id=SOGG6D336JXTX3E5UCE2KKVV32X7L5O4TIB5O6P66N5KUVBFM7IQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.053793783 (570h51m57.924876818s) [noise=true , address=145.79.227.28:4725 , piece_id=UXBAFS7H3AJE7EYEP56Q7KZIZNU6F7MXDAZZOHTBRRXKMKM5727Q, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.054279518 (570h51m57.925362553s) [noise=true , address=45.91.116.12:4226 , piece_id=5COBGM2MUSBHSXQEIA5A45Y22MCVYZ37FKOTUXOZ45SECHCE7VPA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.054631128 (570h51m57.925714163s) [noise=true , address=145.79.227.36:4518 , piece_id=MYK3WULWFFPV2UUOTEGIPWXWX36FL6MDNHWGWNKX2U5YV532WBSQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.054938263 (570h51m57.926021298s) [noise=true , address=145.79.227.39:4015 , piece_id=YH5R23U4C4W5E3ZP7QQBXMNEENVXJZS23YY5UBKFIUE32UEUQZSA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.055269723 (570h51m57.926352769s) [noise=true , address=145.79.227.45:4230 , piece_id=VPB3EGRRWUVLS5I4BGD7WOJ7AG62QU3MFWLZWBHXN3JWFZUUYRNQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.055578111 (570h51m57.926661156s) [noise=true , address=145.79.227.16:4912 , piece_id=27ZKEHEYGLXDKQL4CAZ7J3NK5BJSLYZT4QWDL2EZ7MO6SG2Z7ITQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:59.756060620 (570h51m59.627143655s) [upload=500484]: Next returned nil segment Dec 29 10:20:59.756088703 (570h51m59.627171738s) [upload=500484]: Waiting for error group managing segments... Dec 29 10:21:00.043342830 (570h51m59.914425875s) [noise=true , address=145.79.227.51:4410 , piece_id=BACJKUWO6EF647ZQTEHBC4LBDEUCL5K5VLVQ4Q7RXEVGITK5YC7Q, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.044083620 (570h51m59.915166675s) [noise=true , address=145.79.227.28:4725 , piece_id=UXBAFS7H3AJE7EYEP56Q7KZIZNU6F7MXDAZZOHTBRRXKMKM5727Q, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.046261658 (570h51m59.917344703s) [noise=true , address=145.79.227.36:4518 , piece_id=MYK3WULWFFPV2UUOTEGIPWXWX36FL6MDNHWGWNKX2U5YV532WBSQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.046895204 (570h51m59.917978249s) [noise=true , address=145.79.227.13:4804 , piece_id=FIEA2XHVDBSDA3W7XKQOJGTFODF7GONPTO3E5LOEP4CVERSMCROQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.047755652 (570h51m59.918838687s) [noise=true , address=145.79.227.6:4258 , piece_id=CUNQGOJXPUGWRWHA5TDSXMYFTEZ7KSTMAJQ6RBJFFRK5VUYSPNBA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.049850030 (570h51m59.920933075s) [noise=true , address=145.79.227.39:4015 , piece_id=YH5R23U4C4W5E3ZP7QQBXMNEENVXJZS23YY5UBKFIUE32UEUQZSA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.053312223 (570h51m59.924395268s) [noise=true , address=145.79.227.45:4230 , piece_id=VPB3EGRRWUVLS5I4BGD7WOJ7AG62QU3MFWLZWBHXN3JWFZUUYRNQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.057301489 (570h51m59.928384534s) [noise=true , address=145.79.227.2:4114 , piece_id=COQAGJFIQIDWPL245JNAJJAC4QPS37SNDKBDONAHJX77T4IIB5RQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.062194897 (570h51m59.933277942s) [noise=true , address=145.79.227.47:4265 , piece_id=LKTFJYA4EJWWNH35FXAJRN7XLK2Y5L7UMCCRAURKRU7DSG4E5A4A, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.064783336 (570h51m59.935866381s) [noise=true , address=145.79.227.37:4566 , piece_id=YI4CN6NW7F2QEIKWZG7QOBQVKLYR7K7DX764RH63KLA5W4JQIJVQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.069601421 (570h51m59.940684466s) [noise=true , address=145.79.227.16:4912 , piece_id=27ZKEHEYGLXDKQL4CAZ7J3NK5BJSLYZT4QWDL2EZ7MO6SG2Z7ITQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.322911010 (570h52m0.193994045s) [noise=true , address=45.91.116.14:4191 , piece_id=KCLF74EIYNTZ2QIWN2YDEJLNXJYMKHZYFDF62DLFTZMS54PI3LFQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.351600329 (570h52m0.222683374s) [noise=true , address=45.91.116.8:4047 , piece_id=XKQK4X5YYM4MZI7KA7HDHZHB5IVGDAKAE4IT6SOZRUU4YQB3WYLA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.351628443 (570h52m0.222711478s) [seg_pos={0 0}, upload=500484]: Segment reached optimal threshold of 13 pieces. Dec 29 10:21:00.351838854 (570h52m0.222921899s) [noise=true , address=45.91.116.11:4119 , piece_id=WGNYWLKUJV4BNSJMNFAX3G74KY4UCYEB76QCBORTIGTCQN4DXXKA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 10:21:00.351846017 (570h52m0.222929062s) [noise=true , address=45.91.116.11:4119 , piece_id=WGNYWLKUJV4BNSJMNFAX3G74KY4UCYEB76QCBORTIGTCQN4DXXKA, seg_pos={0 0}, upload=500484]: Long tail cancellation due to optimal threshold Dec 29 10:21:00.352049865 (570h52m0.2231329s) [noise=true , address=45.91.116.12:4226 , piece_id=5COBGM2MUSBHSXQEIA5A45Y22MCVYZ37FKOTUXOZ45SECHCE7VPA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 10:21:00.352056287 (570h52m0.223139322s) [noise=true , address=45.91.116.12:4226 , piece_id=5COBGM2MUSBHSXQEIA5A45Y22MCVYZ37FKOTUXOZ45SECHCE7VPA, seg_pos={0 0}, upload=500484]: Long tail cancellation due to optimal threshold Dec 29 10:21:00.352468202 (570h52m0.223551257s) [noise=true , address=145.79.227.40:4050 , piece_id=SOGG6D336JXTX3E5UCE2KKVV32X7L5O4TIB5O6P66N5KUVBFM7IQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:21:00.352488200 (570h52m0.223571245s) [noise=true , address=145.79.227.40:4050 , piece_id=SOGG6D336JXTX3E5UCE2KKVV32X7L5O4TIB5O6P66N5KUVBFM7IQ, seg_pos={0 0}, upload=500484]: Long tail cancellation due to optimal threshold Dec 29 10:21:00.352520572 (570h52m0.223603607s) [seg_pos={0 0}, upload=500484]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:21:00.352653545 (570h52m0.22373658s) [upload=500484]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:21:00.352661110 (570h52m0.223744145s) [upload=500484]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:21:00.650210069 (570h52m0.521293114s) [upload=500484]: Done uploading segments... Dec 29 10:21:00.650324506 (570h52m0.521407581s) [upload=500484]: Upload finished. err: <nil> Duration: 5.946763366s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.................XXX..XXX......X..X.......X....X............] duration: 3.535991388s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:21:57 UTC) orbiter warning: context deadline exceeded Duration: 3.536020113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..................X......................] duration: 42.005727956s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:21:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 42.005753785s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X......X.X] duration: 30.003642842s trace: https://jaeger.dev.storj.io/trace/71c8c8bce954bac8 FAIL (2025-12-29 10:21:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: XKQK4X5YYM4MZI7KA7HDHZHB5IVGDAKAE4IT6SOZRUU4YQB3WYLA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: KCLF74EIYNTZ2QIWN2YDEJLNXJYMKHZYFDF62DLFTZMS54PI3LFQ) context deadline exceeded ======LOGS====== Dec 29 10:20:57.804358690 (570h51m57.675441725s) [upload=500484]: Starting upload Dec 29 10:20:57.804417613 (570h51m57.675500648s) [upload=500484]: Done starting upload Dec 29 10:20:57.804589891 (570h51m57.675672926s) [upload=500484]: Uploading segments... Dec 29 10:20:57.804623735 (570h51m57.675706771s) [upload=500484]: Got next segment. Inline: false Dec 29 10:20:57.804647822 (570h51m57.675730857s) [upload=500484]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:20:57.804651829 (570h51m57.675734864s) [upload=500484]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:20:58.049104354 (570h51m57.920187389s) [seg_pos={0 0}, upload=500484]: Begin upload segment... Dec 29 10:20:58.049143959 (570h51m57.920227004s) [seg_pos={0 0}, upload=500484]: Done begin upload segment. Dec 29 10:20:58.049169117 (570h51m57.920252152s) [noise=true , address=45.91.116.8:4047 , piece_id=XKQK4X5YYM4MZI7KA7HDHZHB5IVGDAKAE4IT6SOZRUU4YQB3WYLA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.050248312 (570h51m57.921331377s) [noise=true , address=145.79.227.51:4410 , piece_id=BACJKUWO6EF647ZQTEHBC4LBDEUCL5K5VLVQ4Q7RXEVGITK5YC7Q, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.050706204 (570h51m57.921789239s) [noise=true , address=145.79.227.37:4566 , piece_id=YI4CN6NW7F2QEIKWZG7QOBQVKLYR7K7DX764RH63KLA5W4JQIJVQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.051111516 (570h51m57.922194551s) [noise=true , address=145.79.227.2:4114 , piece_id=COQAGJFIQIDWPL245JNAJJAC4QPS37SNDKBDONAHJX77T4IIB5RQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.051569468 (570h51m57.922652503s) [noise=true , address=45.91.116.11:4119 , piece_id=WGNYWLKUJV4BNSJMNFAX3G74KY4UCYEB76QCBORTIGTCQN4DXXKA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.051969198 (570h51m57.923052234s) [noise=true , address=145.79.227.13:4804 , piece_id=FIEA2XHVDBSDA3W7XKQOJGTFODF7GONPTO3E5LOEP4CVERSMCROQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.052297695 (570h51m57.92338073s) [noise=true , address=145.79.227.47:4265 , piece_id=LKTFJYA4EJWWNH35FXAJRN7XLK2Y5L7UMCCRAURKRU7DSG4E5A4A, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.052631500 (570h51m57.923714535s) [noise=true , address=45.91.116.14:4191 , piece_id=KCLF74EIYNTZ2QIWN2YDEJLNXJYMKHZYFDF62DLFTZMS54PI3LFQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.053065376 (570h51m57.924148411s) [noise=true , address=145.79.227.6:4258 , piece_id=CUNQGOJXPUGWRWHA5TDSXMYFTEZ7KSTMAJQ6RBJFFRK5VUYSPNBA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.053458164 (570h51m57.924541199s) [noise=true , address=145.79.227.40:4050 , piece_id=SOGG6D336JXTX3E5UCE2KKVV32X7L5O4TIB5O6P66N5KUVBFM7IQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.053793783 (570h51m57.924876818s) [noise=true , address=145.79.227.28:4725 , piece_id=UXBAFS7H3AJE7EYEP56Q7KZIZNU6F7MXDAZZOHTBRRXKMKM5727Q, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.054279518 (570h51m57.925362553s) [noise=true , address=45.91.116.12:4226 , piece_id=5COBGM2MUSBHSXQEIA5A45Y22MCVYZ37FKOTUXOZ45SECHCE7VPA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.054631128 (570h51m57.925714163s) [noise=true , address=145.79.227.36:4518 , piece_id=MYK3WULWFFPV2UUOTEGIPWXWX36FL6MDNHWGWNKX2U5YV532WBSQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.054938263 (570h51m57.926021298s) [noise=true , address=145.79.227.39:4015 , piece_id=YH5R23U4C4W5E3ZP7QQBXMNEENVXJZS23YY5UBKFIUE32UEUQZSA, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.055269723 (570h51m57.926352769s) [noise=true , address=145.79.227.45:4230 , piece_id=VPB3EGRRWUVLS5I4BGD7WOJ7AG62QU3MFWLZWBHXN3JWFZUUYRNQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:58.055578111 (570h51m57.926661156s) [noise=true , address=145.79.227.16:4912 , piece_id=27ZKEHEYGLXDKQL4CAZ7J3NK5BJSLYZT4QWDL2EZ7MO6SG2Z7ITQ, seg_pos={0 0}, upload=500484]: Uploading piece... Dec 29 10:20:59.756060620 (570h51m59.627143655s) [upload=500484]: Next returned nil segment Dec 29 10:20:59.756088703 (570h51m59.627171738s) [upload=500484]: Waiting for error group managing segments... Dec 29 10:21:00.043342830 (570h51m59.914425875s) [noise=true , address=145.79.227.51:4410 , piece_id=BACJKUWO6EF647ZQTEHBC4LBDEUCL5K5VLVQ4Q7RXEVGITK5YC7Q, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.044083620 (570h51m59.915166675s) [noise=true , address=145.79.227.28:4725 , piece_id=UXBAFS7H3AJE7EYEP56Q7KZIZNU6F7MXDAZZOHTBRRXKMKM5727Q, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.046261658 (570h51m59.917344703s) [noise=true , address=145.79.227.36:4518 , piece_id=MYK3WULWFFPV2UUOTEGIPWXWX36FL6MDNHWGWNKX2U5YV532WBSQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.046895204 (570h51m59.917978249s) [noise=true , address=145.79.227.13:4804 , piece_id=FIEA2XHVDBSDA3W7XKQOJGTFODF7GONPTO3E5LOEP4CVERSMCROQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.047755652 (570h51m59.918838687s) [noise=true , address=145.79.227.6:4258 , piece_id=CUNQGOJXPUGWRWHA5TDSXMYFTEZ7KSTMAJQ6RBJFFRK5VUYSPNBA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.049850030 (570h51m59.920933075s) [noise=true , address=145.79.227.39:4015 , piece_id=YH5R23U4C4W5E3ZP7QQBXMNEENVXJZS23YY5UBKFIUE32UEUQZSA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.053312223 (570h51m59.924395268s) [noise=true , address=145.79.227.45:4230 , piece_id=VPB3EGRRWUVLS5I4BGD7WOJ7AG62QU3MFWLZWBHXN3JWFZUUYRNQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.057301489 (570h51m59.928384534s) [noise=true , address=145.79.227.2:4114 , piece_id=COQAGJFIQIDWPL245JNAJJAC4QPS37SNDKBDONAHJX77T4IIB5RQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.062194897 (570h51m59.933277942s) [noise=true , address=145.79.227.47:4265 , piece_id=LKTFJYA4EJWWNH35FXAJRN7XLK2Y5L7UMCCRAURKRU7DSG4E5A4A, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.064783336 (570h51m59.935866381s) [noise=true , address=145.79.227.37:4566 , piece_id=YI4CN6NW7F2QEIKWZG7QOBQVKLYR7K7DX764RH63KLA5W4JQIJVQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.069601421 (570h51m59.940684466s) [noise=true , address=145.79.227.16:4912 , piece_id=27ZKEHEYGLXDKQL4CAZ7J3NK5BJSLYZT4QWDL2EZ7MO6SG2Z7ITQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.322911010 (570h52m0.193994045s) [noise=true , address=45.91.116.14:4191 , piece_id=KCLF74EIYNTZ2QIWN2YDEJLNXJYMKHZYFDF62DLFTZMS54PI3LFQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.351600329 (570h52m0.222683374s) [noise=true , address=45.91.116.8:4047 , piece_id=XKQK4X5YYM4MZI7KA7HDHZHB5IVGDAKAE4IT6SOZRUU4YQB3WYLA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: <nil> Dec 29 10:21:00.351628443 (570h52m0.222711478s) [seg_pos={0 0}, upload=500484]: Segment reached optimal threshold of 13 pieces. Dec 29 10:21:00.351838854 (570h52m0.222921899s) [noise=true , address=45.91.116.11:4119 , piece_id=WGNYWLKUJV4BNSJMNFAX3G74KY4UCYEB76QCBORTIGTCQN4DXXKA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 29 10:21:00.351846017 (570h52m0.222929062s) [noise=true , address=45.91.116.11:4119 , piece_id=WGNYWLKUJV4BNSJMNFAX3G74KY4UCYEB76QCBORTIGTCQN4DXXKA, seg_pos={0 0}, upload=500484]: Long tail cancellation due to optimal threshold Dec 29 10:21:00.352049865 (570h52m0.2231329s) [noise=true , address=45.91.116.12:4226 , piece_id=5COBGM2MUSBHSXQEIA5A45Y22MCVYZ37FKOTUXOZ45SECHCE7VPA, seg_pos={0 0}, upload=500484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 29 10:21:00.352056287 (570h52m0.223139322s) [noise=true , address=45.91.116.12:4226 , piece_id=5COBGM2MUSBHSXQEIA5A45Y22MCVYZ37FKOTUXOZ45SECHCE7VPA, seg_pos={0 0}, upload=500484]: Long tail cancellation due to optimal threshold Dec 29 10:21:00.352468202 (570h52m0.223551257s) [noise=true , address=145.79.227.40:4050 , piece_id=SOGG6D336JXTX3E5UCE2KKVV32X7L5O4TIB5O6P66N5KUVBFM7IQ, seg_pos={0 0}, upload=500484]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:21:00.352488200 (570h52m0.223571245s) [noise=true , address=145.79.227.40:4050 , piece_id=SOGG6D336JXTX3E5UCE2KKVV32X7L5O4TIB5O6P66N5KUVBFM7IQ, seg_pos={0 0}, upload=500484]: Long tail cancellation due to optimal threshold Dec 29 10:21:00.352520572 (570h52m0.223603607s) [seg_pos={0 0}, upload=500484]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:21:00.352653545 (570h52m0.22373658s) [upload=500484]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:21:00.352661110 (570h52m0.223744145s) [upload=500484]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:21:00.650210069 (570h52m0.521293114s) [upload=500484]: Done uploading segments... Dec 29 10:21:00.650324506 (570h52m0.521407581s) [upload=500484]: Upload finished. err: <nil> Duration: 30.003665596s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..................X.....................] duration: 15.109292093s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:20:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 15.10930617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..................XXX..XXX......X..X.......X....X...........] duration: 3.552031024s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:20:50 UTC) orbiter warning: context deadline exceeded Duration: 3.552061622s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..................X....................] duration: 34.792952157s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:20:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 34.792978426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X......X.] duration: 6.023331915s trace: https://jaeger.dev.storj.io/trace/54bf1973a67e2047 FAIL (2025-12-29 10:20:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:19:02.479792958 (570h50m2.350876003s) [upload=500456]: Starting upload Dec 29 10:19:02.479873751 (570h50m2.350956817s) [upload=500456]: Done starting upload Dec 29 10:19:02.480394603 (570h50m2.351477648s) [upload=500456]: Uploading segments... Dec 29 10:19:02.480530682 (570h50m2.351613748s) [upload=500456]: Got next segment. Inline: false Dec 29 10:19:02.480610103 (570h50m2.351693148s) [upload=500456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:19:02.480617277 (570h50m2.351700322s) [upload=500456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:19:02.726185356 (570h50m2.597268402s) [seg_pos={0 0}, upload=500456]: Begin upload segment... Dec 29 10:19:02.726268414 (570h50m2.597351459s) [seg_pos={0 0}, upload=500456]: Done begin upload segment. Dec 29 10:19:02.726347365 (570h50m2.59743041s) [noise=true , address=77.90.31.13:4626 , piece_id=2IOVWIJ2E3TXNIEL7IPCIUF5TFJ3MH2OXKBYNQU5VPDMMGKWHXPA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.726357564 (570h50m2.59744061s) [noise=true , address=145.79.227.52:4447 , piece_id=Y45FLOLFSWI4AORJV7VHQ52IJL3OSU7DPT27BIETVLGHP6VEGUSA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727048330 (570h50m2.598131375s) [noise=true , address=77.90.31.19:4844 , piece_id=IPK3GB722FOF3AUUJRM2HO2GWZS5DT3JGOCEGAWRHWMEPCWBBMXA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727448341 (570h50m2.598531406s) [noise=true , address=145.79.227.36:4518 , piece_id=4GHJAQ5ROUFUJGZTHQQQLWZML2RYFFNKSKGFM5BRDYLE2HFDDYLQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727680203 (570h50m2.598763238s) [noise=true , address=145.79.227.26:4654 , piece_id=NSD74E2RV22ITZTE5IWV5VJOS3RHZ67A4JDFIHIGRHWYNJGGHVGQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727966648 (570h50m2.599049693s) [noise=true , address=45.91.116.8:4048 , piece_id=6CU47MPB6FG7NI6GZ7IS3OCZND3R7OXOIWUYOZQS2KKYNMNL2WXA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728221954 (570h50m2.599304999s) [noise=true , address=145.79.227.15:4876 , piece_id=3MT2UJ64UI36RO5KW4WDXGQPBCORIXQHQ5XQ5WPBVKPX3MD3X6EQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728415212 (570h50m2.599498258s) [noise=true , address=145.79.227.40:4051 , piece_id=FYR5VDAT5BTMZLIRUQH6JETH2HLGF6F3M26WBIXWUUUHKZAICPAA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728689094 (570h50m2.599772139s) [noise=true , address=145.79.227.19:4078 , piece_id=M4A4EVS2G6VGUVE4QRC3I67XNCMXGTXLZEZOG5D7COGBXXGHQRQQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728942326 (570h50m2.600025371s) [noise=true , address=145.79.227.4:4186 , piece_id=K5BKVLERBV76PKNCRM2UH7FICFGRIQQF5SKZPJMVH5RSCLES6T3Q, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.729381302 (570h50m2.600464347s) [noise=true , address=45.91.116.5:4300 , piece_id=GGDIC3SUGTBGKXHJ43EZK4R5MEAW3D2U5YKSOYXUGU4E37AFRWIQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.729391281 (570h50m2.600474326s) [noise=true , address=45.91.116.12:4227 , piece_id=VVZEE2HYPRGUE37GHNEH4OWIZBSQ4DC5FPVKFHI5JPAMDQAR3PJA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.729912283 (570h50m2.600995328s) [noise=true , address=145.79.227.18:4007 , piece_id=6KV72UE7UJCECXAIFGQUR22URU2AOGSGQN6QPT4XE2ZRP2WCAMNA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.730315571 (570h50m2.601398617s) [noise=true , address=145.79.227.9:4367 , piece_id=N4M527HTJ47UJZZLVPIGBFH7W5KUTSCOES5UR5ZFKITBZQV44FVA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.730505663 (570h50m2.601588708s) [noise=true , address=77.90.31.12:4589 , piece_id=SAQOTJJGBD37IEJ6VRY7S4QCVC4XO3CREAPPE3SYJ3ZJS6Y2IMWQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.731033558 (570h50m2.602116603s) [noise=true , address=145.79.227.38:4978 , piece_id=EJQK5Q5YRBMRDJPLRBQYAKOSOP5452JU2RZOYEPQOPPSTRZYM3BQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:04.431461383 (570h50m4.302544429s) [upload=500456]: Next returned nil segment Dec 29 10:19:04.431554090 (570h50m4.302637145s) [upload=500456]: Waiting for error group managing segments... Dec 29 10:19:04.717685669 (570h50m4.588768704s) [noise=true , address=145.79.227.18:4007 , piece_id=6KV72UE7UJCECXAIFGQUR22URU2AOGSGQN6QPT4XE2ZRP2WCAMNA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.718760545 (570h50m4.589843581s) [noise=true , address=145.79.227.36:4518 , piece_id=4GHJAQ5ROUFUJGZTHQQQLWZML2RYFFNKSKGFM5BRDYLE2HFDDYLQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.732675822 (570h50m4.603758867s) [noise=true , address=145.79.227.9:4367 , piece_id=N4M527HTJ47UJZZLVPIGBFH7W5KUTSCOES5UR5ZFKITBZQV44FVA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.737321468 (570h50m4.608404504s) [noise=true , address=145.79.227.4:4186 , piece_id=K5BKVLERBV76PKNCRM2UH7FICFGRIQQF5SKZPJMVH5RSCLES6T3Q, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.739283654 (570h50m4.6103667s) [noise=true , address=145.79.227.52:4447 , piece_id=Y45FLOLFSWI4AORJV7VHQ52IJL3OSU7DPT27BIETVLGHP6VEGUSA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.741677102 (570h50m4.612760137s) [noise=true , address=145.79.227.15:4876 , piece_id=3MT2UJ64UI36RO5KW4WDXGQPBCORIXQHQ5XQ5WPBVKPX3MD3X6EQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.747612955 (570h50m4.618696s) [noise=true , address=145.79.227.26:4654 , piece_id=NSD74E2RV22ITZTE5IWV5VJOS3RHZ67A4JDFIHIGRHWYNJGGHVGQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.750228906 (570h50m4.621311951s) [noise=true , address=145.79.227.38:4978 , piece_id=EJQK5Q5YRBMRDJPLRBQYAKOSOP5452JU2RZOYEPQOPPSTRZYM3BQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.752485873 (570h50m4.623568909s) [noise=true , address=145.79.227.19:4078 , piece_id=M4A4EVS2G6VGUVE4QRC3I67XNCMXGTXLZEZOG5D7COGBXXGHQRQQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.911265815 (570h50m4.78234885s) [noise=true , address=77.90.31.12:4589 , piece_id=SAQOTJJGBD37IEJ6VRY7S4QCVC4XO3CREAPPE3SYJ3ZJS6Y2IMWQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.038760275 (570h50m4.90984332s) [noise=true , address=77.90.31.19:4844 , piece_id=IPK3GB722FOF3AUUJRM2HO2GWZS5DT3JGOCEGAWRHWMEPCWBBMXA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.071430343 (570h50m4.942513388s) [noise=true , address=45.91.116.8:4048 , piece_id=6CU47MPB6FG7NI6GZ7IS3OCZND3R7OXOIWUYOZQS2KKYNMNL2WXA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.096313204 (570h50m4.967396239s) [noise=true , address=45.91.116.12:4227 , piece_id=VVZEE2HYPRGUE37GHNEH4OWIZBSQ4DC5FPVKFHI5JPAMDQAR3PJA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.096336237 (570h50m4.967419282s) [seg_pos={0 0}, upload=500456]: Segment reached optimal threshold of 13 pieces. Dec 29 10:19:05.096538643 (570h50m4.967621688s) [noise=true , address=45.91.116.5:4300 , piece_id=GGDIC3SUGTBGKXHJ43EZK4R5MEAW3D2U5YKSOYXUGU4E37AFRWIQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:51258->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:51258->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51258->45.91.116.5:4300: use of closed network connection Dec 29 10:19:05.096546077 (570h50m4.967629112s) [noise=true , address=45.91.116.5:4300 , piece_id=GGDIC3SUGTBGKXHJ43EZK4R5MEAW3D2U5YKSOYXUGU4E37AFRWIQ, seg_pos={0 0}, upload=500456]: Long tail cancellation due to optimal threshold Dec 29 10:19:05.096845678 (570h50m4.967928723s) [noise=true , address=77.90.31.13:4626 , piece_id=2IOVWIJ2E3TXNIEL7IPCIUF5TFJ3MH2OXKBYNQU5VPDMMGKWHXPA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 10:19:05.096853092 (570h50m4.967936127s) [noise=true , address=77.90.31.13:4626 , piece_id=2IOVWIJ2E3TXNIEL7IPCIUF5TFJ3MH2OXKBYNQU5VPDMMGKWHXPA, seg_pos={0 0}, upload=500456]: Long tail cancellation due to optimal threshold Dec 29 10:19:05.096956278 (570h50m4.968039323s) [noise=true , address=145.79.227.40:4051 , piece_id=FYR5VDAT5BTMZLIRUQH6JETH2HLGF6F3M26WBIXWUUUHKZAICPAA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 10:19:05.096961678 (570h50m4.968044723s) [noise=true , address=145.79.227.40:4051 , piece_id=FYR5VDAT5BTMZLIRUQH6JETH2HLGF6F3M26WBIXWUUUHKZAICPAA, seg_pos={0 0}, upload=500456]: Long tail cancellation due to optimal threshold Dec 29 10:19:05.096990964 (570h50m4.968074009s) [seg_pos={0 0}, upload=500456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:19:05.097059364 (570h50m4.96814241s) [upload=500456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:19:05.097064334 (570h50m4.968147379s) [upload=500456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:19:05.429733637 (570h50m5.300816672s) [upload=500456]: Done uploading segments... Dec 29 10:19:05.429759507 (570h50m5.300842552s) [upload=500456]: Upload finished. err: <nil> Duration: 6.023366552s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...................XXX..XXX......X..X.......X....X..........] duration: 3.506438024s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:20:08 UTC) orbiter warning: context deadline exceeded Duration: 3.506480215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....................XXX..XXX......X..X.......X....X.........] duration: 3.502283399s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:19:44 UTC) orbiter warning: context deadline exceeded Duration: 3.502311423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X......X] duration: 30.003614882s trace: https://jaeger.dev.storj.io/trace/54bf1973a67e2047 FAIL (2025-12-29 10:19:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:19:02.479792958 (570h50m2.350876003s) [upload=500456]: Starting upload Dec 29 10:19:02.479873751 (570h50m2.350956817s) [upload=500456]: Done starting upload Dec 29 10:19:02.480394603 (570h50m2.351477648s) [upload=500456]: Uploading segments... Dec 29 10:19:02.480530682 (570h50m2.351613748s) [upload=500456]: Got next segment. Inline: false Dec 29 10:19:02.480610103 (570h50m2.351693148s) [upload=500456]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:19:02.480617277 (570h50m2.351700322s) [upload=500456]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:19:02.726185356 (570h50m2.597268402s) [seg_pos={0 0}, upload=500456]: Begin upload segment... Dec 29 10:19:02.726268414 (570h50m2.597351459s) [seg_pos={0 0}, upload=500456]: Done begin upload segment. Dec 29 10:19:02.726347365 (570h50m2.59743041s) [noise=true , address=77.90.31.13:4626 , piece_id=2IOVWIJ2E3TXNIEL7IPCIUF5TFJ3MH2OXKBYNQU5VPDMMGKWHXPA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.726357564 (570h50m2.59744061s) [noise=true , address=145.79.227.52:4447 , piece_id=Y45FLOLFSWI4AORJV7VHQ52IJL3OSU7DPT27BIETVLGHP6VEGUSA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727048330 (570h50m2.598131375s) [noise=true , address=77.90.31.19:4844 , piece_id=IPK3GB722FOF3AUUJRM2HO2GWZS5DT3JGOCEGAWRHWMEPCWBBMXA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727448341 (570h50m2.598531406s) [noise=true , address=145.79.227.36:4518 , piece_id=4GHJAQ5ROUFUJGZTHQQQLWZML2RYFFNKSKGFM5BRDYLE2HFDDYLQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727680203 (570h50m2.598763238s) [noise=true , address=145.79.227.26:4654 , piece_id=NSD74E2RV22ITZTE5IWV5VJOS3RHZ67A4JDFIHIGRHWYNJGGHVGQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.727966648 (570h50m2.599049693s) [noise=true , address=45.91.116.8:4048 , piece_id=6CU47MPB6FG7NI6GZ7IS3OCZND3R7OXOIWUYOZQS2KKYNMNL2WXA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728221954 (570h50m2.599304999s) [noise=true , address=145.79.227.15:4876 , piece_id=3MT2UJ64UI36RO5KW4WDXGQPBCORIXQHQ5XQ5WPBVKPX3MD3X6EQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728415212 (570h50m2.599498258s) [noise=true , address=145.79.227.40:4051 , piece_id=FYR5VDAT5BTMZLIRUQH6JETH2HLGF6F3M26WBIXWUUUHKZAICPAA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728689094 (570h50m2.599772139s) [noise=true , address=145.79.227.19:4078 , piece_id=M4A4EVS2G6VGUVE4QRC3I67XNCMXGTXLZEZOG5D7COGBXXGHQRQQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.728942326 (570h50m2.600025371s) [noise=true , address=145.79.227.4:4186 , piece_id=K5BKVLERBV76PKNCRM2UH7FICFGRIQQF5SKZPJMVH5RSCLES6T3Q, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.729381302 (570h50m2.600464347s) [noise=true , address=45.91.116.5:4300 , piece_id=GGDIC3SUGTBGKXHJ43EZK4R5MEAW3D2U5YKSOYXUGU4E37AFRWIQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.729391281 (570h50m2.600474326s) [noise=true , address=45.91.116.12:4227 , piece_id=VVZEE2HYPRGUE37GHNEH4OWIZBSQ4DC5FPVKFHI5JPAMDQAR3PJA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.729912283 (570h50m2.600995328s) [noise=true , address=145.79.227.18:4007 , piece_id=6KV72UE7UJCECXAIFGQUR22URU2AOGSGQN6QPT4XE2ZRP2WCAMNA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.730315571 (570h50m2.601398617s) [noise=true , address=145.79.227.9:4367 , piece_id=N4M527HTJ47UJZZLVPIGBFH7W5KUTSCOES5UR5ZFKITBZQV44FVA, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.730505663 (570h50m2.601588708s) [noise=true , address=77.90.31.12:4589 , piece_id=SAQOTJJGBD37IEJ6VRY7S4QCVC4XO3CREAPPE3SYJ3ZJS6Y2IMWQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:02.731033558 (570h50m2.602116603s) [noise=true , address=145.79.227.38:4978 , piece_id=EJQK5Q5YRBMRDJPLRBQYAKOSOP5452JU2RZOYEPQOPPSTRZYM3BQ, seg_pos={0 0}, upload=500456]: Uploading piece... Dec 29 10:19:04.431461383 (570h50m4.302544429s) [upload=500456]: Next returned nil segment Dec 29 10:19:04.431554090 (570h50m4.302637145s) [upload=500456]: Waiting for error group managing segments... Dec 29 10:19:04.717685669 (570h50m4.588768704s) [noise=true , address=145.79.227.18:4007 , piece_id=6KV72UE7UJCECXAIFGQUR22URU2AOGSGQN6QPT4XE2ZRP2WCAMNA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.718760545 (570h50m4.589843581s) [noise=true , address=145.79.227.36:4518 , piece_id=4GHJAQ5ROUFUJGZTHQQQLWZML2RYFFNKSKGFM5BRDYLE2HFDDYLQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.732675822 (570h50m4.603758867s) [noise=true , address=145.79.227.9:4367 , piece_id=N4M527HTJ47UJZZLVPIGBFH7W5KUTSCOES5UR5ZFKITBZQV44FVA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.737321468 (570h50m4.608404504s) [noise=true , address=145.79.227.4:4186 , piece_id=K5BKVLERBV76PKNCRM2UH7FICFGRIQQF5SKZPJMVH5RSCLES6T3Q, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.739283654 (570h50m4.6103667s) [noise=true , address=145.79.227.52:4447 , piece_id=Y45FLOLFSWI4AORJV7VHQ52IJL3OSU7DPT27BIETVLGHP6VEGUSA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.741677102 (570h50m4.612760137s) [noise=true , address=145.79.227.15:4876 , piece_id=3MT2UJ64UI36RO5KW4WDXGQPBCORIXQHQ5XQ5WPBVKPX3MD3X6EQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.747612955 (570h50m4.618696s) [noise=true , address=145.79.227.26:4654 , piece_id=NSD74E2RV22ITZTE5IWV5VJOS3RHZ67A4JDFIHIGRHWYNJGGHVGQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.750228906 (570h50m4.621311951s) [noise=true , address=145.79.227.38:4978 , piece_id=EJQK5Q5YRBMRDJPLRBQYAKOSOP5452JU2RZOYEPQOPPSTRZYM3BQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.752485873 (570h50m4.623568909s) [noise=true , address=145.79.227.19:4078 , piece_id=M4A4EVS2G6VGUVE4QRC3I67XNCMXGTXLZEZOG5D7COGBXXGHQRQQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:04.911265815 (570h50m4.78234885s) [noise=true , address=77.90.31.12:4589 , piece_id=SAQOTJJGBD37IEJ6VRY7S4QCVC4XO3CREAPPE3SYJ3ZJS6Y2IMWQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.038760275 (570h50m4.90984332s) [noise=true , address=77.90.31.19:4844 , piece_id=IPK3GB722FOF3AUUJRM2HO2GWZS5DT3JGOCEGAWRHWMEPCWBBMXA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.071430343 (570h50m4.942513388s) [noise=true , address=45.91.116.8:4048 , piece_id=6CU47MPB6FG7NI6GZ7IS3OCZND3R7OXOIWUYOZQS2KKYNMNL2WXA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.096313204 (570h50m4.967396239s) [noise=true , address=45.91.116.12:4227 , piece_id=VVZEE2HYPRGUE37GHNEH4OWIZBSQ4DC5FPVKFHI5JPAMDQAR3PJA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: <nil> Dec 29 10:19:05.096336237 (570h50m4.967419282s) [seg_pos={0 0}, upload=500456]: Segment reached optimal threshold of 13 pieces. Dec 29 10:19:05.096538643 (570h50m4.967621688s) [noise=true , address=45.91.116.5:4300 , piece_id=GGDIC3SUGTBGKXHJ43EZK4R5MEAW3D2U5YKSOYXUGU4E37AFRWIQ, seg_pos={0 0}, upload=500456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:51258->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:51258->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51258->45.91.116.5:4300: use of closed network connection Dec 29 10:19:05.096546077 (570h50m4.967629112s) [noise=true , address=45.91.116.5:4300 , piece_id=GGDIC3SUGTBGKXHJ43EZK4R5MEAW3D2U5YKSOYXUGU4E37AFRWIQ, seg_pos={0 0}, upload=500456]: Long tail cancellation due to optimal threshold Dec 29 10:19:05.096845678 (570h50m4.967928723s) [noise=true , address=77.90.31.13:4626 , piece_id=2IOVWIJ2E3TXNIEL7IPCIUF5TFJ3MH2OXKBYNQU5VPDMMGKWHXPA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 10:19:05.096853092 (570h50m4.967936127s) [noise=true , address=77.90.31.13:4626 , piece_id=2IOVWIJ2E3TXNIEL7IPCIUF5TFJ3MH2OXKBYNQU5VPDMMGKWHXPA, seg_pos={0 0}, upload=500456]: Long tail cancellation due to optimal threshold Dec 29 10:19:05.096956278 (570h50m4.968039323s) [noise=true , address=145.79.227.40:4051 , piece_id=FYR5VDAT5BTMZLIRUQH6JETH2HLGF6F3M26WBIXWUUUHKZAICPAA, seg_pos={0 0}, upload=500456]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 10:19:05.096961678 (570h50m4.968044723s) [noise=true , address=145.79.227.40:4051 , piece_id=FYR5VDAT5BTMZLIRUQH6JETH2HLGF6F3M26WBIXWUUUHKZAICPAA, seg_pos={0 0}, upload=500456]: Long tail cancellation due to optimal threshold Dec 29 10:19:05.096990964 (570h50m4.968074009s) [seg_pos={0 0}, upload=500456]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:19:05.097059364 (570h50m4.96814241s) [upload=500456]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:19:05.097064334 (570h50m4.968147379s) [upload=500456]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:19:05.429733637 (570h50m5.300816672s) [upload=500456]: Done uploading segments... Dec 29 10:19:05.429759507 (570h50m5.300842552s) [upload=500456]: Upload finished. err: <nil> Duration: 30.00364034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..................X...................] duration: 31.334975915s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:19:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 31.335006093s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....................XXX..XXX......X..X.......X....X........] duration: 3.752304869s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:18:56 UTC) orbiter warning: context deadline exceeded Duration: 3.752332772s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....................X..................X..................] duration: 15.215345851s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:18:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 15.215365077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....................X..................X.................] duration: 30.40057562s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:18:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 30.40060194s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X......] duration: 5.672883329s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:18:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 5.672909758s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [......................XXX..XXX......X..X.......X....X.......] duration: 4.686772374s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:18:10 UTC) orbiter warning: context deadline exceeded Duration: 4.686803685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X.....] duration: 5.871649031s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:17:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 5.871675161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.....................X..................X................] duration: 22.121941807s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:17:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 22.121957296s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.......................XXX..XXX......X..X.......X....X......] duration: 3.678425837s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:17:26 UTC) orbiter warning: context deadline exceeded Duration: 3.678454141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.....................X..................X...............] duration: 36.466634318s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:17:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 36.466665739s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X....] duration: 5.668024883s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:17:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 5.668059119s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.....................X..................X..............] duration: 46.196719333s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 46.196800549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [........................XXX..XXX......X..X.......X....X.....] duration: 3.489557626s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:16:32 UTC) orbiter warning: context deadline exceeded Duration: 3.48958615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X...] duration: 6.057012841s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:16:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 6.057029393s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.........................XXX..XXX......X..X.......X....X....] duration: 4.281561202s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:16:05 UTC) orbiter warning: context deadline exceeded Duration: 4.281583013s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X.X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X..] duration: 5.952079063s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:15:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 5.952162491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..........................XXX..XXX......X..X.......X....X...] duration: 3.44309323s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:15:43 UTC) orbiter warning: context deadline exceeded Duration: 3.443196276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X.X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X.] duration: 6.263567768s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:15:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 6.26359469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.....................X..................X.............] duration: 33.093148096s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:15:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 33.093184905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...........................XXX..XXX......X..X.......X....X..] duration: 3.469122912s trace: https://jaeger.dev.storj.io/trace/2b9ec768aab66a85 FAIL (2025-12-29 10:15:01 UTC) orbiter warning: context deadline exceeded Duration: 3.469166514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.X.X.X..........XXXX.X.X...X..X....X...X..XX...X..XX..X] duration: 30.00359338s trace: https://jaeger.dev.storj.io/trace/7acf42fadfa96bf8 FAIL (2025-12-29 10:14:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 10:14:23.732171606 (570h45m23.603254651s) [upload=500388]: Starting upload Dec 29 10:14:23.732234175 (570h45m23.60331721s) [upload=500388]: Done starting upload Dec 29 10:14:23.732285093 (570h45m23.603368138s) [upload=500388]: Uploading segments... Dec 29 10:14:23.732403328 (570h45m23.603486363s) [upload=500388]: Got next segment. Inline: false Dec 29 10:14:23.732430349 (570h45m23.603513384s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 10:14:23.732435639 (570h45m23.603518674s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 10:14:23.979928604 (570h45m23.851011639s) [seg_pos={0 0}, upload=500388]: Begin upload segment... Dec 29 10:14:23.979987837 (570h45m23.851070872s) [seg_pos={0 0}, upload=500388]: Done begin upload segment. Dec 29 10:14:23.980028905 (570h45m23.85111194s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.980110811 (570h45m23.851193856s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981022717 (570h45m23.852105752s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981442065 (570h45m23.85252511s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981580830 (570h45m23.852663865s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.981876593 (570h45m23.852959628s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982015668 (570h45m23.853098703s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982471356 (570h45m23.853554421s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982560254 (570h45m23.853643289s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.982938395 (570h45m23.85402143s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983218548 (570h45m23.854301583s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983376750 (570h45m23.854459795s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983687311 (570h45m23.854770356s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.983798002 (570h45m23.854881037s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984288095 (570h45m23.85537114s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:23.984305438 (570h45m23.855388473s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Uploading piece... Dec 29 10:14:25.677728486 (570h45m25.548811521s) [upload=500388]: Next returned nil segment Dec 29 10:14:25.677756178 (570h45m25.548839223s) [upload=500388]: Waiting for error group managing segments... Dec 29 10:14:25.960756404 (570h45m25.831839439s) [noise=true , address=145.79.227.42:4123 , piece_id=UUZ32FTQHLQQX5RCSEQ7EQUGIK5KCWETULSBVM42LXN3JJZ25IFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.961119225 (570h45m25.83220226s) [noise=true , address=145.79.227.41:4087 , piece_id=OTJZ3EERQV2ED3A4IOAJUKZVLQU627J25IXVY67HZGVDQNNSJFGQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.964360006 (570h45m25.835443041s) [noise=true , address=145.79.227.13:4803 , piece_id=R44OEBUPMCCLYCIHJUTZE3NWHE6DRUTGKLG3IXNLN5MOKAM236VQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.966519648 (570h45m25.837602683s) [noise=true , address=145.79.227.11:4729 , piece_id=AVVDV5ULKC3LHW2MAEWJRHD6LXJVKNTG2LHRIWCHHZ2DAADJQVFQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.976335970 (570h45m25.847419005s) [noise=true , address=145.79.227.4:4186 , piece_id=XS3FYFXSFHWP7VYNXGGJD73M7HVLJICE7SOZG7JVRYCUKDUU7F2A, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986722127 (570h45m25.857805162s) [noise=true , address=145.79.227.26:4654 , piece_id=L3V5W3PJQEAHFCHCFM5FS7RDGIEEBF33O53SLWVSRJQLJHC2OEKQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.986933058 (570h45m25.858016093s) [noise=true , address=145.79.227.15:4875 , piece_id=OXXKBBCEVUM3NTRWF6AERAN33EKZXBLBMRXYLHJGU24WLGFEUDTA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.990794651 (570h45m25.861877686s) [noise=true , address=145.79.227.6:4257 , piece_id=JK33GVFSKE4AKDALW6QMWDF5ZBHDGQXUPPNW3EQBTUP2TNNNTAVA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993421934 (570h45m25.864504969s) [noise=true , address=145.79.227.37:4565 , piece_id=42U2RWRCOS263DLQSXSBJM7OHN4OOPC4M5HTHML4TBKTFE6QLOIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:25.993880608 (570h45m25.864963643s) [noise=true , address=145.79.227.46:4601 , piece_id=XV7M6X7PLQ5QTPJ36RYOHN4Z3TKJF74HBYCMUN2Z5CSZGIX5G3SA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.195649507 (570h45m26.066732542s) [noise=true , address=45.91.116.6:4974 , piece_id=4W22OKT2IAVAFGRQNSCMZSNFFOPMQDJD33MUVNLP43TWD36WH3YQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.233305770 (570h45m26.104388805s) [noise=true , address=45.91.116.13:4156 , piece_id=O74NZMEFYO4ZOI36US7LSPL65SYYATXC76KPXZUIF4QSOVHIQ3OQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339855477 (570h45m26.210938512s) [noise=true , address=77.90.31.12:4590 , piece_id=SDYUXPJSFWV62DAIY7XNFSON4U67ZU3QWTXKEXIB4ZH4USXJCJIA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: <nil> Dec 29 10:14:26.339932053 (570h45m26.211015108s) [seg_pos={0 0}, upload=500388]: Segment reached optimal threshold of 13 pieces. Dec 29 10:14:26.340208419 (570h45m26.211291464s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340217145 (570h45m26.21130019s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL): protocol: expected piece hash; context canceled; EOF Dec 29 10:14:26.340225402 (570h45m26.211308437s) [noise=true , address=145.79.227.14:4838 , piece_id=HQJF3TIKDWE7PIL5OPMWREUKXDHLLAK3FO2XKDINSXFHTTAPJXGA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340469557 (570h45m26.211552612s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40404->77.90.31.11:4554: use of closed network connection Dec 29 10:14:26.340479035 (570h45m26.21156207s) [noise=true , address=77.90.31.11:4554 , piece_id=ALGE6ZGMBY4QYEFUMZ3C3JYVQITUNI56YNW3BXOVXFSKFJJSKTCQ, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340491899 (570h45m26.211574934s) [noise=true , address=145.79.227.40:4050 , piece_id=SZSRDO6XYHWQCI2PVO7R43B2Q7KKUD3AYVVGEPNRQ6X47PRJ65OA, seg_pos={0 0}, upload=500388]: Long tail cancellation due to optimal threshold Dec 29 10:14:26.340567152 (570h45m26.211650187s) [seg_pos={0 0}, upload=500388]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 10:14:26.340766281 (570h45m26.211849326s) [upload=500388]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:14:26.340774016 (570h45m26.211857061s) [upload=500388]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:14:26.640395542 (570h45m26.511478587s) [upload=500388]: Done uploading segments... Dec 29 10:14:26.640413216 (570h45m26.511496261s) [upload=500388]: Upload finished. err: <nil> Duration: 30.003616853s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.....................X..................X............] duration: 30.121626787s trace: https://jaeger.dev.storj.io/trace/1c2841f8b76ec97 FAIL (2025-12-29 10:14:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660036239 (570h34m56.531119274s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660557591 (570h34m56.531640636s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.660904843 (570h34m56.531987888s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661173535 (570h34m56.53225658s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661427067 (570h34m56.532510102s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.661719615 (570h34m56.53280265s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662051867 (570h34m56.533134902s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662162207 (570h34m56.533245242s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662493889 (570h34m56.533576934s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662598839 (570h34m56.533681884s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662752230 (570h34m56.533835275s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.662936571 (570h34m56.534019606s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663244698 (570h34m56.534327733s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.663671922 (570h34m56.534754957s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:56.664054861 (570h34m56.535137906s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Uploading piece... Dec 29 10:03:58.018415867 (570h34m57.889498902s) [upload=500239]: Next returned nil segment Dec 29 10:03:58.018438900 (570h34m57.889521945s) [upload=500239]: Waiting for error group managing segments... Dec 29 10:03:58.515039410 (570h34m58.386122455s) [noise=true , address=145.79.227.10:4427 , piece_id=4I7IXAKNXRV55KER5QBCUNERHTDTSGSMYAA2GHIN7TLASFFASN5A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.555770938 (570h34m58.426853983s) [noise=true , address=145.79.227.34:4918 , piece_id=JIA7YFDQKUK4W2FJMNDJGEH3BVMGV6MNEKWBCBAB5T36FLJ6KSHA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.651193109 (570h34m58.522276144s) [noise=true , address=145.79.227.17:4002 , piece_id=6YUIM6ZPTUZDXTZGNWZJTXAA5TP67PL6FNEENHMCL7QGMVHJELMQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.739888609 (570h34m58.610971654s) [noise=true , address=145.79.227.40:4063 , piece_id=BEJXBZUDIMWTIZN4EO25HB6NWSJAPTHYBIXEWYFRFGN33LJEU4TA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.809258360 (570h34m58.680341405s) [noise=true , address=77.90.31.12:4593 , piece_id=ZVXBLLLG55336XYE26XZOZ6PKIO5LLVF5U2M44LCEAAGF7KU6G3A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.825226948 (570h34m58.696309993s) [noise=true , address=77.90.31.11:4585 , piece_id=RIY6OFAU24BBHBBVLQHSJNT2LZPDYQPV535NYUCM3K2ZOTTLF76Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.854740285 (570h34m58.72582333s) [noise=true , address=145.79.227.29:4953 , piece_id=3LDYKZLFCGGSHMRATAS4Z4L6W2RKQGYENKYROYHNHBDJ2RJPMJGA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:58.929971287 (570h34m58.801054322s) [noise=true , address=77.90.31.16:4766 , piece_id=47KCPG2T4WBX73LAKUICIN4XKGQJCR2ELYB45GGSLPJQTE2XLUWA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.025312452 (570h34m58.896395497s) [noise=true , address=77.90.31.14:4671 , piece_id=OSHK43BZYYWUV3OVV5D4DOFFCEG3JVL3JUC6FEEU7K3ES2CNQPOQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.247439288 (570h34m59.118522323s) [noise=true , address=193.149.164.143:4701 , piece_id=KF77J74MEZIFPO63M3EINYADKSBSBP7JCD76SOM2RHAQRAO3NQMA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.302692559 (570h34m59.173775595s) [noise=true , address=45.91.116.8:4058 , piece_id=BBB46IXEQC7DXXQCEO7IAXC7GVZJVFXOTHSNKQJT3OSU35J7ZFYA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.397583217 (570h34m59.268666252s) [noise=true , address=145.79.227.44:4216 , piece_id=NPNYW3ALFSRIGIFBF64HKHV2ZGROMZITWWFEAJPQNUOB4GWHKD6A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.412665988 (570h34m59.283749023s) [noise=true , address=145.79.227.35:4515 , piece_id=TAMP6Q4MP6NDNPT3AUZRLPDHYDAAO6W7TTYL4L6NLU4CYGWUR5YQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427694143 (570h34m59.298777188s) [noise=true , address=145.79.227.30:4769 , piece_id=HXGZEP5V5QTI446ZQDZDAW7N6ZHCBHGT2ZPWSYZVKIJQXVJ2RIEQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: <nil> Dec 29 10:03:59.427714282 (570h34m59.298797317s) [seg_pos={0 0}, upload=500239]: Segment reached optimal threshold of 14 pieces. Dec 29 10:03:59.427916867 (570h34m59.298999912s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.427923861 (570h34m59.299006916s) [noise=true , address=145.79.227.43:4180 , piece_id=IZKJ7QZQZS5YOU4FXDIUNFOQJWYT6D2JJYKQCFWQOFWF32JB3T6Q, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428083565 (570h34m59.2991666s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T24rtYBJ47E6Ef1ZZKMWcpBrWBpCg5QV2ZHniCZZkzJF2yUNk): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428098924 (570h34m59.299181959s) [noise=true , address=145.79.227.49:4360 , piece_id=4C2NYKDQ6YGXFSQLFO6OQQQDDO5Y5NFJIBXYIQPIEKNNN5R5XE2A, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428109444 (570h34m59.299192489s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: expected piece hash; context canceled; EOF Dec 29 10:03:59.428116497 (570h34m59.299199542s) [noise=true , address=64.49.13.27:4966 , piece_id=Q6UBKCAYWO6WOVXLOIUWJO2WWENIVXLID2WWF74LDVMUDENYHUDA, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428318442 (570h34m59.299401487s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39602->64.49.13.169:4713: use of closed network connection Dec 29 10:03:59.428326026 (570h34m59.299409061s) [noise=true , address=64.49.13.169:4713 , piece_id=U5ATF2TBML47QNPJUNNYXKX2N3AD6UBHTPKVSRXSLC7LQ2EK23ZQ, seg_pos={0 0}, upload=500239]: Long tail cancellation due to optimal threshold Dec 29 10:03:59.428345895 (570h34m59.29942893s) [seg_pos={0 0}, upload=500239]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 10:03:59.428447809 (570h34m59.299530844s) [upload=500239]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 10:03:59.428453139 (570h34m59.299536184s) [upload=500239]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 10:03:59.718471291 (570h34m59.589554336s) [upload=500239]: Done uploading segments... Dec 29 10:03:59.718493734 (570h34m59.589576769s) [upload=500239]: Upload finished. err: <nil> Duration: 30.121673407s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)