Orbiter History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...............X..........................X............X....] duration: 6.365397487s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:56:13 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 6.365424249s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.698315668s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:56:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.698335274s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 4.728296364s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:55:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.728316722s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 15.36023023s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:55:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 15.360251922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.62152474s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:55:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621542244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.668625977s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:55:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.668650223s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 4.656816038s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:55:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.656840004s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [................X..........................X............X...] duration: 4.111762901s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:55:01 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 4.111786716s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.00072041s trace: https://jaeger.dev.storj.io/trace/5e3081a3454ea1e7 FAIL (2026-05-26 03:54:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000745258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.................X..........................X............X..] duration: 3.96998691s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:54:19 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 3.970012598s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.78687614s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:54:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.786892092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.678961337s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:54:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.678983811s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 4.703741003s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:54:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.703769648s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [..................X..........................X............X.] duration: 4.05129585s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:53:44 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 4.051308935s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [...................X..........................X............X] duration: 1m0.001982852s trace: https://jaeger.dev.storj.io/trace/19211c2f74202db8 FAIL (2026-05-26 03:53:40 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.690928799 (25h49m3.474268898s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695665447 (25h49m3.479005546s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.695751701 (25h49m3.4790918s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697094423 (25h49m3.480434522s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.697638391 (25h49m3.48097849s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.698665050 (25h49m3.482005149s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.699785328 (25h49m3.483125427s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.702752419 (25h49m3.486092518s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.704092555 (25h49m3.487432654s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.705138702 (25h49m3.488478801s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.706185118 (25h49m3.489525217s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.707114050 (25h49m3.49045415s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.708141521 (25h49m3.49148162s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.709078178 (25h49m3.492418277s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:17.710062877 (25h49m3.493402976s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Uploading piece... May 26 03:53:18.760650563 (25h49m4.543990662s) [upload=24702]: Next returned nil segment May 26 03:53:18.760676152 (25h49m4.544016251s) [upload=24702]: Waiting for error group managing segments... May 26 03:53:19.039059242 (25h49m4.822399341s) [noise=true , address=212.102.52.215:4379 , piece_id=7E6CKA5WJ3VYZW3PKV5DARVJMWJSVCDCJZEQ3RP6MEUIGA527IRA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.047478717 (25h49m4.830818816s) [noise=true , address=212.102.52.214:4352 , piece_id=FBWTTC2ZANORNYJPJ2YDLKXZLUYKN6E27WGERYCMZGMVLPDKQTVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.048580760 (25h49m4.831920859s) [noise=true , address=212.102.52.217:4423 , piece_id=HUUZCOIPQJLOGIIZRUJATM4DWF6NVQOGXPYTX2WSYAIAUZMXYIUA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.050451389 (25h49m4.833791488s) [noise=true , address=212.102.52.213:4322 , piece_id=5B4OT45N6242R4C6L2XUMAUI35DFMLQ3APNBC6RAY7HON3XVAL7Q, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.051314045 (25h49m4.834654144s) [noise=true , address=212.102.52.216:4412 , piece_id=CEBIYSCGY5NNIZWUO6UCISSC5ISXJYQR743CMJKS4MFPREETKELQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056328372 (25h49m4.839668461s) [noise=true , address=212.102.52.214:4348 , piece_id=3IWQG7VIIY5DRFQQOGDOCSI4NHQRIRYKLDYDEOUL4STXEX7J45KA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.056847763 (25h49m4.840187862s) [noise=true , address=212.102.52.216:4415 , piece_id=AUKJDHGJVWZFEGSIHUX3LUDDYIRT7FVOXNBC3XWYYIWDWQSHX2OQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.058311306 (25h49m4.841651405s) [noise=true , address=212.102.52.213:4323 , piece_id=NMWWZSAEUFZ7D7R3C7ESAD6ENMMM5IQ2G5GVWKYECCZPOEWZMAWQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.059487700 (25h49m4.8428278s) [noise=true , address=212.102.52.217:4427 , piece_id=6MIPVORBS7XU5CR3C6MNBEPU2PSEGNXEJ3X5EMZ3VPIIZQFLDFHQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.060932807 (25h49m4.844272906s) [noise=true , address=212.102.52.213:4326 , piece_id=722JTEO7PCLPY5JJGQTJLGX7AMGDAMZQM4ZOH6J4NDPKJ3MS72WA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.070400593 (25h49m4.853740692s) [noise=true , address=212.102.52.217:4432 , piece_id=KB5SF4FPKKFXWZRWF75JLYFQE5JDHQHCLSEXAD3ZOD7P2ICL2HVQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.075073779 (25h49m4.858413878s) [noise=true , address=212.102.52.213:4330 , piece_id=ZY3VH73B6ZUONSDP3N6X7LVDUBVQ6VK6VEUPJ2375YLYQZ3EL2OA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.076258709 (25h49m4.859598818s) [noise=true , address=212.102.52.217:4429 , piece_id=DWSXRXT5XMLO57OEL6X2UT4TFFHIK5KUIIGKKBDT2NV53ONNEAMQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.079504452 (25h49m4.862844551s) [noise=true , address=212.102.52.215:4377 , piece_id=HY7OR65ZZ26OB7LDQVNLVIVPPWTEJ4XK3JBIK56N74DUVRULORSQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.080733237 (25h49m4.864073336s) [noise=true , address=212.102.52.215:4389 , piece_id=KICYFDFUM2I2N57C5REEY5R5VDN3HM5EIAKQEOBDMHXHB676FXPQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081099115 (25h49m4.864439214s) [noise=true , address=212.102.52.216:4404 , piece_id=PZ4RYIBQIPKTQMBIVKNH7TRYWVBK37IDGZCT257YOR4E7HYUUGQA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: <nil> May 26 03:53:19.081106920 (25h49m4.864447009s) [seg_pos={0 0}, upload=24702]: Satisfied cohort requirements after 16 pieces May 26 03:53:19.081289148 (25h49m4.864629247s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1286SqXMbtXfUYTRS8gDRUVFNfWEimRFetY3Z7WgZCgBwxox9R6): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081294578 (25h49m4.864634677s) [noise=true , address=212.102.52.214:4354 , piece_id=ELSGHXMWTFMABFFGQ62ZM6OKW5ELXETLW6IFTMAZGBGEI6VSMY7A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081334153 (25h49m4.864674273s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081342860 (25h49m4.864682959s) [noise=true , address=212.102.52.216:4393 , piece_id=M7MMB3MIU6RNTVXGT2FING2ARWLQB32DPXBLOPLVKMJUW4YUWT6A, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081455605 (25h49m4.864795704s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081464271 (25h49m4.864804391s) [noise=true , address=212.102.52.215:4385 , piece_id=23MU2HSF2GM27V43H76TPTDWOJVGDATAUZVTAMTBMKPY777KXFOQ, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081469111 (25h49m4.864809211s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:53:19.081474802 (25h49m4.864814901s) [noise=true , address=212.102.52.214:4347 , piece_id=T5IOD3YWLMECVSIHVTM4CTLK5J6ODZKFU3Q7Q5QACETZ6GHRLGNA, seg_pos={0 0}, upload=24702]: Long tail cancellation due to optimal threshold May 26 03:53:19.081501393 (25h49m4.864841492s) [seg_pos={0 0}, upload=24702]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:53:19.081607916 (25h49m4.864948015s) [upload=24702]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:53:19.081612945 (25h49m4.864953044s) [upload=24702]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:53:37.504968733 (25h49m23.288308832s) [upload=24702]: Done uploading segments... May 26 03:53:37.504991115 (25h49m23.288331214s) [upload=24702]: Upload finished. err: <nil> Duration: 1m0.002000777s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 13.377190937s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:53:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.377215876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.637564405s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:53:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.637590065s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 4.669850542s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:52:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.669871993s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 4.6903643s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:52:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.690383288s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [....................X..........................X............] duration: 4.080868503s trace: https://jaeger.dev.storj.io/trace/602f8fde42281540 FAIL (2026-05-26 03:52:20 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2STGQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801389929 (25h39m13.584730028s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801601543 (25h39m13.584941642s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801652901 (25h39m13.58499299s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801925932 (25h39m13.585266031s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802015393 (25h39m13.585355492s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802145120 (25h39m13.585485219s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802299164 (25h39m13.585639263s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802351864 (25h39m13.585691964s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802558038 (25h39m13.585898137s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802569590 (25h39m13.585909689s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802803687 (25h39m13.586143786s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803036250 (25h39m13.586376349s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803265137 (25h39m13.586605236s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803510785 (25h39m13.586850884s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803730054 (25h39m13.587070153s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:28.871532967 (25h39m14.654873066s) [upload=24536]: Next returned nil segment May 26 03:43:28.871543608 (25h39m14.654883707s) [upload=24536]: Waiting for error group managing segments... May 26 03:43:29.153559348 (25h39m14.936899457s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.153821548 (25h39m14.937161647s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.156587746 (25h39m14.939927845s) [noise=true , address=212.102.52.216:4412 , piece_id=2LT4CBKYKRZQFP3DJWEA2ISV3UKE22JJOKE4ZVAFAEAYHJ5N2QYQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.164039414 (25h39m14.947379513s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.166133019 (25h39m14.949473118s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170002502 (25h39m14.953342601s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170778232 (25h39m14.954118331s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176294627 (25h39m14.959634727s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176630518 (25h39m14.959970617s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182717552 (25h39m14.966057651s) [noise=true , address=212.102.52.214:4348 , piece_id=4TMT7IO6BZBUAD3CJSTMZY3AFCX64VAKQIUUQQNM52WU5K62STGQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182761005 (25h39m14.966101094s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.183151079 (25h39m14.966491178s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.195333794 (25h39m14.978673893s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.201022619 (25h39m14.984362718s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.210522314 (25h39m14.993862413s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211512484 (25h39m14.994852573s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211530097 (25h39m14.994870186s) [seg_pos={0 0}, upload=24536]: Satisfied cohort requirements after 16 pieces May 26 03:43:29.211717494 (25h39m14.995057593s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgavD3HvXpxXs2PV5NacdCzKYqkBXv2oLPquUyDFDk7S93brT): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211723186 (25h39m14.995063285s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211748603 (25h39m14.995088703s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211756119 (25h39m14.995096218s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211861930 (25h39m14.995202029s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211867390 (25h39m14.995207489s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211886617 (25h39m14.995226716s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211893119 (25h39m14.995233218s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211909291 (25h39m14.99524939s) [seg_pos={0 0}, upload=24536]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:43:29.211968233 (25h39m14.995308332s) [upload=24536]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:43:29.211972351 (25h39m14.99531245s) [upload=24536]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:43:34.790160469 (25h39m20.573500568s) [upload=24536]: Done uploading segments... May 26 03:43:34.790177321 (25h39m20.57351742s) [upload=24536]: Upload finished. err: <nil> Duration: 4.080884985s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 8.625925163s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:52:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.625939252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 4.683666038s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:52:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.683691386s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.622870986s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:51:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.622890403s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..........................X...........] duration: 5.23959546s trace: https://jaeger.dev.storj.io/trace/602f8fde42281540 FAIL (2026-05-26 03:51:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2STGQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801389929 (25h39m13.584730028s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801601543 (25h39m13.584941642s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801652901 (25h39m13.58499299s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801925932 (25h39m13.585266031s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802015393 (25h39m13.585355492s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802145120 (25h39m13.585485219s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802299164 (25h39m13.585639263s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802351864 (25h39m13.585691964s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802558038 (25h39m13.585898137s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802569590 (25h39m13.585909689s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802803687 (25h39m13.586143786s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803036250 (25h39m13.586376349s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803265137 (25h39m13.586605236s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803510785 (25h39m13.586850884s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803730054 (25h39m13.587070153s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:28.871532967 (25h39m14.654873066s) [upload=24536]: Next returned nil segment May 26 03:43:28.871543608 (25h39m14.654883707s) [upload=24536]: Waiting for error group managing segments... May 26 03:43:29.153559348 (25h39m14.936899457s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.153821548 (25h39m14.937161647s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.156587746 (25h39m14.939927845s) [noise=true , address=212.102.52.216:4412 , piece_id=2LT4CBKYKRZQFP3DJWEA2ISV3UKE22JJOKE4ZVAFAEAYHJ5N2QYQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.164039414 (25h39m14.947379513s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.166133019 (25h39m14.949473118s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170002502 (25h39m14.953342601s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170778232 (25h39m14.954118331s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176294627 (25h39m14.959634727s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176630518 (25h39m14.959970617s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182717552 (25h39m14.966057651s) [noise=true , address=212.102.52.214:4348 , piece_id=4TMT7IO6BZBUAD3CJSTMZY3AFCX64VAKQIUUQQNM52WU5K62STGQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182761005 (25h39m14.966101094s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.183151079 (25h39m14.966491178s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.195333794 (25h39m14.978673893s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.201022619 (25h39m14.984362718s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.210522314 (25h39m14.993862413s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211512484 (25h39m14.994852573s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211530097 (25h39m14.994870186s) [seg_pos={0 0}, upload=24536]: Satisfied cohort requirements after 16 pieces May 26 03:43:29.211717494 (25h39m14.995057593s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgavD3HvXpxXs2PV5NacdCzKYqkBXv2oLPquUyDFDk7S93brT): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211723186 (25h39m14.995063285s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211748603 (25h39m14.995088703s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211756119 (25h39m14.995096218s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211861930 (25h39m14.995202029s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211867390 (25h39m14.995207489s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211886617 (25h39m14.995226716s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211893119 (25h39m14.995233218s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211909291 (25h39m14.99524939s) [seg_pos={0 0}, upload=24536]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:43:29.211968233 (25h39m14.995308332s) [upload=24536]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:43:29.211972351 (25h39m14.99531245s) [upload=24536]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:43:34.790160469 (25h39m20.573500568s) [upload=24536]: Done uploading segments... May 26 03:43:34.790177321 (25h39m20.57351742s) [upload=24536]: Upload finished. err: <nil> Duration: 5.239620917s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 4.714082744s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:51:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.714118724s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.645693368s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:51:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.645713356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 18.04295445s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:51:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 18.042976242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [......................X..........................X..........] duration: 4.098032919s trace: https://jaeger.dev.storj.io/trace/602f8fde42281540 FAIL (2026-05-26 03:51:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2STGQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801389929 (25h39m13.584730028s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801601543 (25h39m13.584941642s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801652901 (25h39m13.58499299s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801925932 (25h39m13.585266031s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802015393 (25h39m13.585355492s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802145120 (25h39m13.585485219s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802299164 (25h39m13.585639263s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802351864 (25h39m13.585691964s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802558038 (25h39m13.585898137s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802569590 (25h39m13.585909689s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802803687 (25h39m13.586143786s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803036250 (25h39m13.586376349s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803265137 (25h39m13.586605236s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803510785 (25h39m13.586850884s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803730054 (25h39m13.587070153s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:28.871532967 (25h39m14.654873066s) [upload=24536]: Next returned nil segment May 26 03:43:28.871543608 (25h39m14.654883707s) [upload=24536]: Waiting for error group managing segments... May 26 03:43:29.153559348 (25h39m14.936899457s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.153821548 (25h39m14.937161647s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.156587746 (25h39m14.939927845s) [noise=true , address=212.102.52.216:4412 , piece_id=2LT4CBKYKRZQFP3DJWEA2ISV3UKE22JJOKE4ZVAFAEAYHJ5N2QYQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.164039414 (25h39m14.947379513s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.166133019 (25h39m14.949473118s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170002502 (25h39m14.953342601s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170778232 (25h39m14.954118331s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176294627 (25h39m14.959634727s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176630518 (25h39m14.959970617s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182717552 (25h39m14.966057651s) [noise=true , address=212.102.52.214:4348 , piece_id=4TMT7IO6BZBUAD3CJSTMZY3AFCX64VAKQIUUQQNM52WU5K62STGQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182761005 (25h39m14.966101094s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.183151079 (25h39m14.966491178s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.195333794 (25h39m14.978673893s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.201022619 (25h39m14.984362718s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.210522314 (25h39m14.993862413s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211512484 (25h39m14.994852573s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211530097 (25h39m14.994870186s) [seg_pos={0 0}, upload=24536]: Satisfied cohort requirements after 16 pieces May 26 03:43:29.211717494 (25h39m14.995057593s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgavD3HvXpxXs2PV5NacdCzKYqkBXv2oLPquUyDFDk7S93brT): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211723186 (25h39m14.995063285s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211748603 (25h39m14.995088703s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211756119 (25h39m14.995096218s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211861930 (25h39m14.995202029s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211867390 (25h39m14.995207489s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211886617 (25h39m14.995226716s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211893119 (25h39m14.995233218s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211909291 (25h39m14.99524939s) [seg_pos={0 0}, upload=24536]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:43:29.211968233 (25h39m14.995308332s) [upload=24536]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:43:29.211972351 (25h39m14.99531245s) [upload=24536]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:43:34.790160469 (25h39m20.573500568s) [upload=24536]: Done uploading segments... May 26 03:43:34.790177321 (25h39m20.57351742s) [upload=24536]: Upload finished. err: <nil> Duration: 4.098088984s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.633593707s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:51:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.633607793s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.685425546s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:50:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.685453029s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.207161808s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.207188036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.197233823s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:50:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.197256366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..........................X.........] duration: 4.240129694s trace: https://jaeger.dev.storj.io/trace/602f8fde42281540 FAIL (2026-05-26 03:50:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2STGQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801389929 (25h39m13.584730028s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801601543 (25h39m13.584941642s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801652901 (25h39m13.58499299s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801925932 (25h39m13.585266031s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802015393 (25h39m13.585355492s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802145120 (25h39m13.585485219s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802299164 (25h39m13.585639263s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802351864 (25h39m13.585691964s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802558038 (25h39m13.585898137s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802569590 (25h39m13.585909689s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802803687 (25h39m13.586143786s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803036250 (25h39m13.586376349s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803265137 (25h39m13.586605236s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803510785 (25h39m13.586850884s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803730054 (25h39m13.587070153s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:28.871532967 (25h39m14.654873066s) [upload=24536]: Next returned nil segment May 26 03:43:28.871543608 (25h39m14.654883707s) [upload=24536]: Waiting for error group managing segments... May 26 03:43:29.153559348 (25h39m14.936899457s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.153821548 (25h39m14.937161647s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.156587746 (25h39m14.939927845s) [noise=true , address=212.102.52.216:4412 , piece_id=2LT4CBKYKRZQFP3DJWEA2ISV3UKE22JJOKE4ZVAFAEAYHJ5N2QYQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.164039414 (25h39m14.947379513s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.166133019 (25h39m14.949473118s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170002502 (25h39m14.953342601s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170778232 (25h39m14.954118331s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176294627 (25h39m14.959634727s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176630518 (25h39m14.959970617s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182717552 (25h39m14.966057651s) [noise=true , address=212.102.52.214:4348 , piece_id=4TMT7IO6BZBUAD3CJSTMZY3AFCX64VAKQIUUQQNM52WU5K62STGQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182761005 (25h39m14.966101094s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.183151079 (25h39m14.966491178s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.195333794 (25h39m14.978673893s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.201022619 (25h39m14.984362718s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.210522314 (25h39m14.993862413s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211512484 (25h39m14.994852573s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211530097 (25h39m14.994870186s) [seg_pos={0 0}, upload=24536]: Satisfied cohort requirements after 16 pieces May 26 03:43:29.211717494 (25h39m14.995057593s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgavD3HvXpxXs2PV5NacdCzKYqkBXv2oLPquUyDFDk7S93brT): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211723186 (25h39m14.995063285s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211748603 (25h39m14.995088703s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211756119 (25h39m14.995096218s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211861930 (25h39m14.995202029s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211867390 (25h39m14.995207489s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211886617 (25h39m14.995226716s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211893119 (25h39m14.995233218s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211909291 (25h39m14.99524939s) [seg_pos={0 0}, upload=24536]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:43:29.211968233 (25h39m14.995308332s) [upload=24536]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:43:29.211972351 (25h39m14.99531245s) [upload=24536]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:43:34.790160469 (25h39m20.573500568s) [upload=24536]: Done uploading segments... May 26 03:43:34.790177321 (25h39m20.57351742s) [upload=24536]: Upload finished. err: <nil> Duration: 4.240150395s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 4.675144067s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:50:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.675164936s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 9.412802856s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:50:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 9.412827293s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 4.717465946s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:49:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.717485103s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.686863016s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:49:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.6868863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.236151563s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:49:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.236173294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [........................X..........................X........] duration: 4.000368279s trace: https://jaeger.dev.storj.io/trace/602f8fde42281540 FAIL (2026-05-26 03:49:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2STGQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801389929 (25h39m13.584730028s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801601543 (25h39m13.584941642s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801652901 (25h39m13.58499299s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801925932 (25h39m13.585266031s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802015393 (25h39m13.585355492s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802145120 (25h39m13.585485219s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802299164 (25h39m13.585639263s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802351864 (25h39m13.585691964s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802558038 (25h39m13.585898137s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802569590 (25h39m13.585909689s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802803687 (25h39m13.586143786s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803036250 (25h39m13.586376349s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803265137 (25h39m13.586605236s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803510785 (25h39m13.586850884s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803730054 (25h39m13.587070153s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:28.871532967 (25h39m14.654873066s) [upload=24536]: Next returned nil segment May 26 03:43:28.871543608 (25h39m14.654883707s) [upload=24536]: Waiting for error group managing segments... May 26 03:43:29.153559348 (25h39m14.936899457s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.153821548 (25h39m14.937161647s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.156587746 (25h39m14.939927845s) [noise=true , address=212.102.52.216:4412 , piece_id=2LT4CBKYKRZQFP3DJWEA2ISV3UKE22JJOKE4ZVAFAEAYHJ5N2QYQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.164039414 (25h39m14.947379513s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.166133019 (25h39m14.949473118s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170002502 (25h39m14.953342601s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170778232 (25h39m14.954118331s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176294627 (25h39m14.959634727s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176630518 (25h39m14.959970617s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182717552 (25h39m14.966057651s) [noise=true , address=212.102.52.214:4348 , piece_id=4TMT7IO6BZBUAD3CJSTMZY3AFCX64VAKQIUUQQNM52WU5K62STGQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182761005 (25h39m14.966101094s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.183151079 (25h39m14.966491178s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.195333794 (25h39m14.978673893s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.201022619 (25h39m14.984362718s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.210522314 (25h39m14.993862413s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211512484 (25h39m14.994852573s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211530097 (25h39m14.994870186s) [seg_pos={0 0}, upload=24536]: Satisfied cohort requirements after 16 pieces May 26 03:43:29.211717494 (25h39m14.995057593s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgavD3HvXpxXs2PV5NacdCzKYqkBXv2oLPquUyDFDk7S93brT): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211723186 (25h39m14.995063285s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211748603 (25h39m14.995088703s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211756119 (25h39m14.995096218s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211861930 (25h39m14.995202029s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211867390 (25h39m14.995207489s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211886617 (25h39m14.995226716s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211893119 (25h39m14.995233218s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211909291 (25h39m14.99524939s) [seg_pos={0 0}, upload=24536]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:43:29.211968233 (25h39m14.995308332s) [upload=24536]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:43:29.211972351 (25h39m14.99531245s) [upload=24536]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:43:34.790160469 (25h39m20.573500568s) [upload=24536]: Done uploading segments... May 26 03:43:34.790177321 (25h39m20.57351742s) [upload=24536]: Upload finished. err: <nil> Duration: 4.000389401s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 4.716947309s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:49:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.71697335s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.405502335s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:48:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.4055218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.654703158s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:48:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.654726193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.489517453s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:48:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.489539385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 4.681879721s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:48:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.681919808s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.802282065s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:48:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.802296011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 4.746198242s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:48:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.746222097s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..........................X.......] duration: 6.453258195s trace: https://jaeger.dev.storj.io/trace/602f8fde42281540 FAIL (2026-05-26 03:48:15 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2STGQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801389929 (25h39m13.584730028s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801601543 (25h39m13.584941642s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801652901 (25h39m13.58499299s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.801925932 (25h39m13.585266031s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802015393 (25h39m13.585355492s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802145120 (25h39m13.585485219s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802299164 (25h39m13.585639263s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802351864 (25h39m13.585691964s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802558038 (25h39m13.585898137s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802569590 (25h39m13.585909689s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.802803687 (25h39m13.586143786s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803036250 (25h39m13.586376349s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803265137 (25h39m13.586605236s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803510785 (25h39m13.586850884s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:27.803730054 (25h39m13.587070153s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Uploading piece... May 26 03:43:28.871532967 (25h39m14.654873066s) [upload=24536]: Next returned nil segment May 26 03:43:28.871543608 (25h39m14.654883707s) [upload=24536]: Waiting for error group managing segments... May 26 03:43:29.153559348 (25h39m14.936899457s) [noise=true , address=212.102.52.217:4436 , piece_id=FGLPX5UIE2NBEGGNJOLONXHL2NQZXLFN2GMMRWUKJZRV4FZVXD7A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.153821548 (25h39m14.937161647s) [noise=true , address=212.102.52.217:4438 , piece_id=JDQ7SC3P7CGTFFE6E3DCQAIT4GIWQO7YMBU2EY7KI6CE7D3E4W5Q, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.156587746 (25h39m14.939927845s) [noise=true , address=212.102.52.216:4412 , piece_id=2LT4CBKYKRZQFP3DJWEA2ISV3UKE22JJOKE4ZVAFAEAYHJ5N2QYQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.164039414 (25h39m14.947379513s) [noise=true , address=212.102.52.215:4389 , piece_id=3P47A77S2DVB5IGGFQB3Z2MXDALPWKV7FOQQ64FFTVAWA63FQ34A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.166133019 (25h39m14.949473118s) [noise=true , address=212.102.52.216:4404 , piece_id=OF2Y4HDWWDM5OGAH5A3JRI37CSBH7VISBLG4J75N5HOGMEIEFLRA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170002502 (25h39m14.953342601s) [noise=true , address=212.102.52.215:4377 , piece_id=JDLE7BD3L5KIR6Z7STDXXVJ3VZ2MMXKXGMYX54DAEOKYQ47CMZUQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.170778232 (25h39m14.954118331s) [noise=true , address=212.102.52.213:4330 , piece_id=R2OPGSEGD3UWOVQEXZLQ7Y23G7VJS4BG4TFYNIJCIT473YRFV3PQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176294627 (25h39m14.959634727s) [noise=true , address=212.102.52.214:4354 , piece_id=FGWUTAWDM4KYSQ4RPX4QNOE6STLTYRDC2OXG5KXJ3SGA72V4S76A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.176630518 (25h39m14.959970617s) [noise=true , address=212.102.52.214:4352 , piece_id=5RDFVLFOEQ6T67X6GXTP7E3VSZNJ4CJZ7K7A56QT6QLFR5EHPKBQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182717552 (25h39m14.966057651s) [noise=true , address=212.102.52.214:4348 , piece_id=4TMT7IO6BZBUAD3CJSTMZY3AFCX64VAKQIUUQQNM52WU5K62STGQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.182761005 (25h39m14.966101094s) [noise=true , address=212.102.52.217:4424 , piece_id=V3V4EGAZ7UGRYCNNWGKICAIUF7UROSRIUZL7DMABVBW2KZL53CBA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.183151079 (25h39m14.966491178s) [noise=true , address=212.102.52.213:4323 , piece_id=D5TA5CTBN56DT6OARCXYV33A6D2DQQYNN46AQ2E26MGK3MLF6FZQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.195333794 (25h39m14.978673893s) [noise=true , address=212.102.52.213:4326 , piece_id=D2ABGWTPMA74DPNJTBRQCBLV7MCHL6AJVEXTOMCRCRW7LWG5D65A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.201022619 (25h39m14.984362718s) [noise=true , address=212.102.52.213:4341 , piece_id=NI6WWO2KSFST7KYYL6H2DW6WS7CWEQ62NL6FLX43BDZY5KBMCM2A, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.210522314 (25h39m14.993862413s) [noise=true , address=212.102.52.215:4379 , piece_id=ZYKHE7MQXFLEITWJFUS7WWZP4EX7LO6Z3NE62HTPRHDEVSJCYITQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211512484 (25h39m14.994852573s) [noise=true , address=212.102.52.217:4429 , piece_id=F6WQCGMJU6RTF2T74JLDU63H6WRNWKNOBOEMAY2TTK7LXTJSLTJA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: <nil> May 26 03:43:29.211530097 (25h39m14.994870186s) [seg_pos={0 0}, upload=24536]: Satisfied cohort requirements after 16 pieces May 26 03:43:29.211717494 (25h39m14.995057593s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fgavD3HvXpxXs2PV5NacdCzKYqkBXv2oLPquUyDFDk7S93brT): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211723186 (25h39m14.995063285s) [noise=true , address=212.102.52.216:4415 , piece_id=LCRQM4CHBJ47PNA2HZJB6ZWMHA32K54SQRXMD33KTB252XBMR2GQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211748603 (25h39m14.995088703s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GobdbvLireyEB6WtiokfHU1PnDtmyoGTZctB7PTzcjG7Spz1z): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211756119 (25h39m14.995096218s) [noise=true , address=212.102.52.216:4393 , piece_id=NYYKFHYWUPYRP4BXEG3ITD2EP45GRT3GWYNE7WEWJTL3EPVC5KUA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211861930 (25h39m14.995202029s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MxFYgP6Z65sARb62qQQtY5PYemzq42o2CBrEgDeDZevgHjpwK): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211867390 (25h39m14.995207489s) [noise=true , address=212.102.52.215:4385 , piece_id=6QONEVSEQ2RTB3FFQ7AZO3PQE4OIOYVQARJJMXBBSY7W224GRRTQ, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211886617 (25h39m14.995226716s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127vua8xA3HHNDurSeqQEtNeQ7R4WJz2kTegBasccKVGrshF1dY): protocol: expected piece hash; context canceled; EOF May 26 03:43:29.211893119 (25h39m14.995233218s) [noise=true , address=212.102.52.214:4347 , piece_id=SA2ONHZ5G2I2J2QQYDRJWH2FA6OXE4TBKOIPK6FKB2E4IIEMHTOA, seg_pos={0 0}, upload=24536]: Long tail cancellation due to optimal threshold May 26 03:43:29.211909291 (25h39m14.99524939s) [seg_pos={0 0}, upload=24536]: Done waiting for segment. successful: 16 optimal: 16 errs: <nil> May 26 03:43:29.211968233 (25h39m14.995308332s) [upload=24536]: Flush batch item: pb.BatchRequestItem_SegmentCommit May 26 03:43:29.211972351 (25h39m14.99531245s) [upload=24536]: Flush batch item: pb.BatchRequestItem_ObjectCommit May 26 03:43:34.790160469 (25h39m20.573500568s) [upload=24536]: Done uploading segments... May 26 03:43:34.790177321 (25h39m20.57351742s) [upload=24536]: Upload finished. err: <nil> Duration: 6.453292131s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.703123934s trace: https://jaeger.dev.storj.io/trace/58499abd97a4a547 FAIL (2026-05-26 03:47:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 26 but got 15) ======LOGS====== [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:07.482929983 (25h28m53.266270072s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Uploading piece... May 26 03:33:08.324361378 (25h28m54.107701477s) [upload=24372]: Next returned nil segment May 26 03:33:08.324374614 (25h28m54.107714703s) [upload=24372]: Waiting for error group managing segments... May 26 03:33:08.731031075 (25h28m54.514371174s) [noise=true , address=81.171.14.70:4039 , piece_id=GFPLM7V2D4I2XSTWNIU3322NH4RNAD33QI3GVP3N7WDWG3DUU64Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731274119 (25h28m54.514614218s) [noise=true , address=81.171.14.70:4026 , piece_id=ZQLJIJPHJ2KP65YOMMZNILTNR6WYYHSVSPKBU4WKFTOQFNI3VCVQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.731699240 (25h28m54.515039339s) [noise=true , address=81.171.14.70:4048 , piece_id=7B5HNGMTIYVRA3YBRXPX5UN4YLKA47SA3VEA6V5VKTE5HKNKRXZQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.740303036 (25h28m54.523643126s) [noise=true , address=81.171.14.70:4033 , piece_id=YI6PF2NM7LLPZZ42TCKBLGTHJNPJADH5XUURGJI4G2ITEIASGCFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.746923018 (25h28m54.530263107s) [noise=true , address=81.171.14.70:4042 , piece_id=WXRNES7VUHBWFEBLVPJFHOMCB36C35BRZOPIDKY37EC3XL272H4Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:08.787794853 (25h28m54.571134952s) [noise=true , address=81.171.14.70:4041 , piece_id=GALIU56QSLRKVUA4PPLGYWFR7TLDCHJMWM6ZFSYXBIZE2T3NKE5A, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.223068517 (25h28m55.006408616s) [noise=true , address=81.171.14.70:4044 , piece_id=JUVFW2SHNIMCED7Y4S22YJQ246TIUUOY2SCF3HYMOQNKDV2ORPUA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.230623161 (25h28m55.01396326s) [noise=true , address=81.171.14.70:4043 , piece_id=NMZQKARJH6BVDK6HCQTR7AUTY6SFTQM7J5FA37ZYNMEPY3PBPZTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.231318919 (25h28m55.014659018s) [noise=true , address=89.149.201.147:4011 , piece_id=HRINJWZ55AT43T7YKNCXU3WJIGLOD6WS63YV4NNL6AYS35XRCXFA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.234455043 (25h28m55.017795142s) [noise=true , address=89.149.201.147:4021 , piece_id=QM5OZORHMNRUTV7BNXUZN66UKFN3FU4HBJ7JJ6BY42GAR65SVDMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.250660443 (25h28m55.034000542s) [noise=true , address=89.149.201.147:4017 , piece_id=MXYGBV5XQEM3RWWSWWWVNJMGH2UJGT2IFPDSNQX5DPWGGACKJHTA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.280697715 (25h28m55.064037814s) [noise=true , address=89.149.201.147:4009 , piece_id=Q2FKJYIANL7NYOY4F7JGJK6SU5MUSLYUZ7FGWG26CFTA4FHJXGSQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.305574250 (25h28m55.088914349s) [noise=true , address=89.149.201.147:4020 , piece_id=BWDGFP3HHL4DJTNA4RZHGHKJKSKIRKNXQCXFKLA3CQNC56IW7CRA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.326442828 (25h28m55.109782927s) [noise=true , address=89.149.201.147:4014 , piece_id=Z3ONVFREJGQVKARJMVY3XDZ5DFQUYCG32I2MYSRXYHPVWGHBDCVA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:33:09.371048036 (25h28m55.154388125s) [noise=true , address=89.149.201.147:4010 , piece_id=7WWA7ZEMEEVSLO2A43ABF72GBXM7K7RFU4AIBRCUZKN2Z6ICSGPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: <nil> May 26 03:34:06.805550190 (25h29m52.58889029s) [noise=true , address=185.167.204.205:4027 , piece_id=OXLJIIPWJHYL6JJMFP3A7554T4IZK6MCFNDVEY4QWSVEXJ6TIQYA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1CVzMfvFN4cYHWnSVrXBmHjLR9KtGqsMimwVq1QJzjgpbjeszV, address:185.167.204.205:4027): protocol: rpc: dial tcp 185.167.204.205:4027: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805635323 (25h29m52.588975472s) [noise=true , address=185.167.204.206:4050 , piece_id=4F7GNQFFMNVYG3NEN6WYMFCKSQJJHPAWF6X654TY7NKJXQHZZ5TA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1UR3Amy4kaFW5xhpD1s4WAVW15smyz4up1EZ5SH5Eg126SnAaz, address:185.167.204.206:4050): protocol: rpc: dial tcp 185.167.204.206:4050: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805730364 (25h29m52.589070463s) [noise=true , address=185.167.204.205:4035 , piece_id=E6EQRUEBXNBLSY7MYHBU4ZLMASLFMHW6EYQV7XCJGBCWUGSLCFFQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805915377 (25h29m52.589255466s) [noise=true , address=185.167.204.205:4014 , piece_id=BBV7YAA34BVWKO3MDENOBCNZRPI7OR62OL35D3XPHXZ66IDAUWPQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12ajStqnU4gCGh2ffTPhrFAP9UtB8ybC9Y4pmgnHfx5hK3rVk8q, address:185.167.204.205:4014): protocol: rpc: dial tcp 185.167.204.205:4014: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.805932159 (25h29m52.589272258s) [noise=true , address=185.167.204.205:4002 , piece_id=NGMYOKOP5QTTUSLIAJHRB43TLBYQFRCSBM6U26BZBXLKA2ENTIQA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806082816 (25h29m52.589422915s) [noise=true , address=185.167.204.206:4065 , piece_id=EIGTOJDLRXTIY2HATT22TIN2TF7M7UIXZDMRSMFMYBHHTLHE5KBA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12oPo26KZGZr7o7M18KZiX5PMAAMwHK8FPT59y1hSLhkB6z3Lcj, address:185.167.204.206:4065): protocol: rpc: dial tcp 185.167.204.206:4065: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806120208 (25h29m52.589460327s) [noise=true , address=185.167.204.206:4072 , piece_id=XR5UOQJMCKXCZ7BUFKO3HLS57RPG6SLH2S2RSO4KEE7PBAL5US3Q, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:17uNR1rhasAqhitGStEMyeW7EmHrz7R87jr5nJB7S7CbZvy59v, address:185.167.204.206:4072): protocol: rpc: dial tcp 185.167.204.206:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806248693 (25h29m52.589588792s) [noise=true , address=185.167.204.205:4023 , piece_id=PQFGRQSVJ4ZLC5GYQTOXIURXJRPODTLPK36BQISX66L7LUC5ZSBQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:41840->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806423337 (25h29m52.589763426s) [noise=true , address=185.167.204.205:4021 , piece_id=HYN6CWTC3KLSDUQUZ2AWWFB4TVY5SPH4SMT3VHM62NAXHVFH6JEA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12FDxcFVTdQeudwXi3bRkqnBSQ7NhjChgqkCZLmEC9T4bxdGA4w, address:185.167.204.205:4021): protocol: rpc: dial tcp 185.167.204.205:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806471758 (25h29m52.589811857s) [noise=true , address=185.167.204.206:4056 , piece_id=HT6UEZP55TVKRHY2NEWXYXYC456BSVYHVJ2PIAOSICUDH5UZDKMQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1uSTuNA28oH2s5Rkv2jaKoC2nZWSgpMJXyxwFTCKbr44EvEXrC, address:185.167.204.206:4056): protocol: rpc: dial tcp 185.167.204.206:4056: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806497608 (25h29m52.589837707s) [noise=true , address=185.167.204.206:4041 , piece_id=G3NXN2TLCIFTC4B7FFL7VLHFVKWDNLBXA6BLRDDKAWRKUZE34BYQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:1m3rag6G75ZAJkXs6tMaHceq1e2mxp2LrYW8zct4p1Ra3pNhaC, address:185.167.204.206:4041): protocol: rpc: dial tcp 185.167.204.206:4041: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:34:06.806627034 (25h29m52.589967134s) [noise=true , address=185.167.204.206:4052 , piece_id=2OU6DVHU6K4HUJJ6K6I52DYAC5XPBNBGZFFJ4LAAYJZLYZEM26FA, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:12Xr3UmaAcshmxZMecu5i7VHA6LJxiZm9u6wngapEma3hz3tNmn, address:185.167.204.206:4052): protocol: rpc: dial tcp 185.167.204.206:4052: i/o timeout; context deadline exceeded May 26 03:34:06.806767703 (25h29m52.590107792s) [noise=true , address=185.167.204.206:4063 , piece_id=GZKUBYQZSIFP53QKGBI3R3V52SEY7AWHEL5CBVMTWWSIHCP5HKEQ, seg_pos={0 0}, upload=24372]: Done uploading piece. err: ecclient: upload failed (node:16qB6oqhjMfxCSDZJers7bCt7Eeomd3njuz1qR5F9PTpuoHX5U, address:185.167.204.206:4063): protocol: context deadline exceeded; context deadline exceeded May 26 03:34:06.806787201 (25h29m52.5901273s) [seg_pos={0 0}, upload=24372]: Done waiting for segment. successful: 15 optimal: 26 errs: <nil> May 26 03:34:06.983344969 (25h29m52.766685058s) [upload=24372]: Done uploading segments... May 26 03:34:06.983374684 (25h29m52.766714783s) [upload=24372]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 15) Duration: 4.70313796s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.340398909s trace: https://jaeger.dev.storj.io/trace/2f99fbcd176b1617 FAIL (2026-05-26 03:47:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== May 26 03:37:44.554844381 (25h33m30.33818448s) [upload=24452]: Starting upload May 26 03:37:44.554914775 (25h33m30.338254884s) [upload=24452]: Done starting upload May 26 03:37:44.555010087 (25h33m30.338350186s) [upload=24452]: Uploading segments... May 26 03:37:44.555154713 (25h33m30.338494812s) [upload=24452]: Got next segment. Inline: false May 26 03:37:44.555190892 (25h33m30.338530991s) [upload=24452]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:37:44.555194178 (25h33m30.338534277s) [upload=24452]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:37:44.796393493 (25h33m30.579733593s) [seg_pos={0 0}, upload=24452]: Begin upload segment... May 26 03:37:44.796434813 (25h33m30.579774902s) [seg_pos={0 0}, upload=24452]: Done begin upload segment. May 26 03:37:44.796455061 (25h33m30.57979516s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796479167 (25h33m30.579819266s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796547607 (25h33m30.579887706s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.796944846 (25h33m30.580284935s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797283402 (25h33m30.580623501s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.797625474 (25h33m30.580965573s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798009868 (25h33m30.581349967s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798332895 (25h33m30.581672994s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.798639480 (25h33m30.581979569s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799006550 (25h33m30.582346649s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.799936012 (25h33m30.583276102s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.801075256 (25h33m30.584415356s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.802249427 (25h33m30.585589526s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.803442773 (25h33m30.586782872s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.804531642 (25h33m30.587871741s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:44.805927776 (25h33m30.589267875s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Uploading piece... May 26 03:37:46.520818854 (25h33m32.304158953s) [upload=24452]: Next returned nil segment May 26 03:37:46.520830196 (25h33m32.304170295s) [upload=24452]: Waiting for error group managing segments... May 26 03:37:46.802301060 (25h33m32.585641159s) [noise=true , address=145.79.227.8:4332 , piece_id=LRJG7OY6SCUWAEATGPICV3SQLID25AAPL5DJHMRX4T66MOVT4ARQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.805062139 (25h33m32.588402238s) [noise=true , address=145.79.227.12:4769 , piece_id=4CMAB7IQW3DY72X4R2M4D3GEDZXHQHWYEVBN3VKXMSSYABFOMJWA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818069497 (25h33m32.601409586s) [noise=true , address=145.79.227.49:4340 , piece_id=H4PP2BCFBCRUPINAFKPEIS7IKRCDVILC4EPYEEWNTBSHTNICWGGA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.818763521 (25h33m32.60210362s) [noise=true , address=145.79.227.26:4657 , piece_id=JCI2K722O5F2HGZR4POW3UMUCGPXQUHMSZFGJL7IL2TSY3T4LKNA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.832663342 (25h33m32.616003451s) [noise=true , address=145.79.227.21:4473 , piece_id=CV3E5SXKUA3OAW2QRQHHNDSK74MYME6AUNE7YD2YQJMP6MAVSQEQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.841016390 (25h33m32.624356489s) [noise=true , address=145.79.227.45:4234 , piece_id=2YBD3GONCSIP3A63LSVRI647YC3NTWSXUCBY5ZWEIWMT5B7D7YLQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.842493007 (25h33m32.625833106s) [noise=true , address=145.79.227.40:4049 , piece_id=BAAQV3TB3GMDZNLG6UUEMPNJ2R5E5RVSEFBTOPDQHNUXJFB6XOLA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.855901301 (25h33m32.63924139s) [noise=true , address=145.79.227.5:4224 , piece_id=QYELKAAUIN72JYBUBRWGKGOF5D4FZ6M4CV2LIRWYMX6Q3FU2TECA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.872794464 (25h33m32.656134563s) [noise=true , address=145.79.227.10:4401 , piece_id=FDK7SAUVXCQM2XHWJQMLHMLEVYVYH3FWGR4CI5KLFPPFZJRCNJKQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:46.904359019 (25h33m32.687699148s) [noise=true , address=145.79.227.15:4873 , piece_id=MCX4ZRVNDVT53WE7W5X7JZ53ZFVOJIUWMZBPHCXAZBUXS4RULCTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.599588789 (25h33m33.382928888s) [noise=true , address=45.91.116.12:4227 , piece_id=VUZQ6SGI6MGKMHZOT4KZP2Z2CTUGHTQ6373WQRZPTN2KANTEGDTA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:37:47.616657837 (25h33m33.399997936s) [noise=true , address=45.91.116.4:4947 , piece_id=D2SJUTVUCPKJ2UL2UVZPCGITIXAIMYGDVSSD3Z5XUUN7SB6WAADA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: <nil> May 26 03:38:13.890981361 (25h33m59.67432147s) [noise=true , address=77.90.31.12:4589 , piece_id=P6TPHQWR7IOYUKZ4VZKC6CH77YAJJUU2A33WTECMQPUVI2ZCUQYA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: rpc: dial tcp 77.90.31.12:4589: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891042047 (25h33m59.674382146s) [noise=true , address=77.90.31.16:4735 , piece_id=EHMPUNQ3X4KQ3OS3Y2ZSAN6QLRGPS2W6DL635MO7IN6FO7QX7LFQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891153340 (25h33m59.674493439s) [noise=true , address=77.90.31.17:4773 , piece_id=45PHIJTVNU5KTCWK4RUCJ3MF2TJSA4B37W4CG6UJ3GFRRPQQ5WZA, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ, address:77.90.31.17:4773): protocol: context deadline exceeded; context deadline exceeded May 26 03:38:13.891203274 (25h33m59.674543393s) [noise=true , address=77.90.31.11:4558 , piece_id=SGKGDSAPYDNZ2KDMXJXCQTENMFPAGI6VAXZY3WQUBIZSRFZHPWZQ, seg_pos={0 0}, upload=24452]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: write tcp 188.166.225.170:40140->77.90.31.11:4558: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:38:13.891231057 (25h33m59.674571156s) [seg_pos={0 0}, upload=24452]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> May 26 03:38:14.130774583 (25h33m59.914114683s) [upload=24452]: Done uploading segments... May 26 03:38:14.130801014 (25h33m59.914141113s) [upload=24452]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.340414598s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 13.424268593s trace: https://jaeger.dev.storj.io/trace/bb228e6f4e56e8d FAIL (2026-05-26 03:47:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== May 26 03:34:51.057311479 (25h30m36.840651579s) [upload=24400]: Starting upload May 26 03:34:51.057391622 (25h30m36.840731721s) [upload=24400]: Done starting upload May 26 03:34:51.057490661 (25h30m36.84083076s) [upload=24400]: Uploading segments... May 26 03:34:51.057591854 (25h30m36.840931953s) [upload=24400]: Got next segment. Inline: false May 26 03:34:51.057625859 (25h30m36.840965958s) [upload=24400]: Flush batch item: pb.BatchRequestItem_ObjectBegin May 26 03:34:51.057629526 (25h30m36.840969625s) [upload=24400]: Flush batch item: pb.BatchRequestItem_SegmentBegin May 26 03:34:51.292929294 (25h30m37.076269403s) [seg_pos={0 0}, upload=24400]: Begin upload segment... May 26 03:34:51.292964903 (25h30m37.076305002s) [seg_pos={0 0}, upload=24400]: Done begin upload segment. May 26 03:34:51.292989609 (25h30m37.076329708s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293001382 (25h30m37.076341471s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293425101 (25h30m37.0767652s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.293751414 (25h30m37.077091513s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294126739 (25h30m37.077466838s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294459584 (25h30m37.077799673s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.294747314 (25h30m37.078087413s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295038469 (25h30m37.078378558s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295304607 (25h30m37.078644706s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295608777 (25h30m37.078948876s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.295911264 (25h30m37.079251363s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296369418 (25h30m37.079709517s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.296709377 (25h30m37.080049476s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297572744 (25h30m37.080912833s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.297731818 (25h30m37.081071917s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:51.298580286 (25h30m37.081920385s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Uploading piece... May 26 03:34:52.567697680 (25h30m38.351037769s) [upload=24400]: Next returned nil segment May 26 03:34:52.567710364 (25h30m38.351050453s) [upload=24400]: Waiting for error group managing segments... May 26 03:34:52.916105923 (25h30m38.699446022s) [noise=true , address=64.49.13.170:4720 , piece_id=3JSZ3RY72MI6B5UZ4YF573TDEXJ7YHY5CSIQCBLWWANB4VQELJWA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.926296076 (25h30m38.709636175s) [noise=true , address=193.149.164.78:4477 , piece_id=KYNLJJXC6KSJ2PVMHFPX32PULM4KX62KTGUYLOGKTVBB4QRKUBGA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:52.979491645 (25h30m38.762831744s) [noise=true , address=64.49.13.27:4966 , piece_id=P5KKTYHEQHYFSX3NUPZYLWPANMGVZQ5TEFWUDVM2MNRARCALCQEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.040935621 (25h30m38.82427572s) [noise=true , address=193.149.164.79:4519 , piece_id=TUMGIPTR4S5DH52NXSYJ4XO2J4NY32MMDYNGCCQFEZWKQGGKPXMA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.166520140 (25h30m38.949860239s) [noise=true , address=145.79.227.16:4933 , piece_id=FYYQJFTVO6HZ3DOT6P5RPEG5BY2SWXL6NJND4J5JA3MZMOXDJZXA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.178543722 (25h30m38.961883831s) [noise=true , address=145.79.227.15:4886 , piece_id=TDZGTW5XAY556AK6TSF77NSO5SZ4NMSCXKA7UWJI5JP5JD77GHNA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.186434207 (25h30m38.969774316s) [noise=true , address=145.79.227.6:4290 , piece_id=QFMX4XC7KKPVRS2I5BY2A7J4I36PSPV6TUID2BYU5DSZD5BW6JBA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.246791750 (25h30m39.030131849s) [noise=true , address=145.79.227.27:4713 , piece_id=WQCHEO37RTEZPPLOC2VODLMSCHMQJIKAWVQXOMRFPIAWQBW5ICEQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.253734938 (25h30m39.037075037s) [noise=true , address=145.79.227.17:4978 , piece_id=RMAJF4V52KZ3I22H7LO36T3UY56KCPRDAGJQFDD4MN3WPPDUQ4EQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:53.545641886 (25h30m39.328981985s) [noise=true , address=167.17.72.69:4728 , piece_id=7E2HBBOSORYIC252ZUCYRZZE7GYYASJOX4ISTYYOXOKDEWBCSBYA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.536476788 (25h30m40.319816887s) [noise=true , address=45.91.116.13:4172 , piece_id=4TDNGUKRNXUSAM5VIHMN3WKGPINTAFBKGW2TQHLB2EJSQYHC562Q, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.731478628 (25h30m40.514818727s) [noise=true , address=45.91.116.11:4146 , piece_id=LNPP32ULINK5DMOGMYXQHIMFBZ3WK4E2FMSGZBDHQ4JIIO277PGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:34:54.814909195 (25h30m40.598249294s) [noise=true , address=167.17.72.71:4781 , piece_id=RNZYCCOYUWBRED5CYLWAYN45JE2YPMN5NBEB37QWIBJXSUJINXCA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: <nil> May 26 03:35:50.395667765 (25h31m36.179007854s) [noise=true , address=77.90.31.11:4579 , piece_id=ISHWIRMXXBRLGEFTOOLWVO5LFMJSPRLG4FTZBQ5QYIVCZMSKEDGQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12dRcuBLuT2ybWGzdkp4vyo7e37qrZk6SBpWWHty5WKRoVHp1p, address:77.90.31.11:4579): protocol: write tcp 188.166.225.170:40706->77.90.31.11:4579: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395764269 (25h31m36.179104358s) [noise=true , address=77.90.31.10:4523 , piece_id=HKAHCRV6A56O4IVDNCH7XLA6F46LJWG5AQVOO6VSC2A44FJETQOA, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: rpc: dial tcp 77.90.31.10:4523: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395792703 (25h31m36.179132802s) [noise=true , address=77.90.31.16:4741 , piece_id=5R4PRRZH23RXIAR2OW6UEMSEZSSNJRGTMLUPS7XPUNOWIKDPJCPQ, seg_pos={0 0}, upload=24400]: Done uploading piece. err: ecclient: upload failed (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR, address:77.90.31.16:4741): protocol: write tcp 188.166.225.170:45908->77.90.31.16:4741: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op May 26 03:35:50.395808893 (25h31m36.179148992s) [seg_pos={0 0}, upload=24400]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> May 26 03:35:50.632114722 (25h31m36.415454841s) [upload=24400]: Done uploading segments... May 26 03:35:50.632169156 (25h31m36.415509255s) [upload=24400]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.424297247s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)