Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.211316709s FAIL (2025-10-06 22:30:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.21134881s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.082847291s FAIL (2025-10-06 22:30:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.082880955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.507285415s FAIL (2025-10-06 22:30:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.507333547s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.029001867s FAIL (2025-10-06 22:29:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.02917655s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.460052095s FAIL (2025-10-06 22:29:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.460085699s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.972556284s FAIL (2025-10-06 22:29:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 5.972584338s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.348522272s FAIL (2025-10-06 22:29:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.348554714s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.761791866s FAIL (2025-10-06 22:28:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.761821733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.411740953s FAIL (2025-10-06 22:28:22 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.411774395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.172317738s FAIL (2025-10-06 22:28:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.172346673s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.415204116s FAIL (2025-10-06 22:27:50 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.415227871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.766892003s FAIL (2025-10-06 22:27:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.766914987s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.683057499s FAIL (2025-10-06 22:27:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.683128796s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.312863974s FAIL (2025-10-06 22:26:55 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.312891806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.498518581s FAIL (2025-10-06 22:26:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.498547586s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.623750988s FAIL (2025-10-06 22:26:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.623783891s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.293599462s FAIL (2025-10-06 22:26:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.293617265s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.241684495s FAIL (2025-10-06 22:26:11 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.241720574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.775451771s FAIL (2025-10-06 22:26:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775493892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.305285415s FAIL (2025-10-06 22:25:51 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.305319661s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.675748498s FAIL (2025-10-06 22:25:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.675777836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.388970141s FAIL (2025-10-06 22:25:08 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.389006242s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.078874275s FAIL (2025-10-06 22:25:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.078906415s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.157916013s FAIL (2025-10-06 22:25:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.157936713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.43141792s FAIL (2025-10-06 22:24:52 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.431450522s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.169216763s FAIL (2025-10-06 22:24:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.169248213s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.558995565s FAIL (2025-10-06 22:24:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.559031363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.763770342s FAIL (2025-10-06 22:23:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.763799978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.303787882s FAIL (2025-10-06 22:23:48 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.303813301s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.288889645s FAIL (2025-10-06 22:23:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.288989866s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.119458023s FAIL (2025-10-06 22:23:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.119484763s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.496115567s FAIL (2025-10-06 22:23:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.496145804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.994658901s FAIL (2025-10-06 22:23:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 5.994698828s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.624595794s FAIL (2025-10-06 22:22:49 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.624626442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 6.37326593s FAIL (2025-10-06 22:22:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.373295056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.494131429s FAIL (2025-10-06 22:21:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.494167237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.074100363s FAIL (2025-10-06 22:21:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.074141311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.400472534s FAIL (2025-10-06 22:21:21 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.400497662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.788000538s FAIL (2025-10-06 22:21:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.788029393s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.18522892s FAIL (2025-10-06 22:21:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.185263767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.33656403s FAIL (2025-10-06 22:20:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.336582426s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 6.770281058s FAIL (2025-10-06 22:20:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.770306575s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.406422653s FAIL (2025-10-06 22:20:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.406456157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.383881071s FAIL (2025-10-06 22:20:16 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.383916598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.825804611s FAIL (2025-10-06 22:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82584595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.410157323s FAIL (2025-10-06 22:19:40 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.410182742s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.804130601s FAIL (2025-10-06 22:19:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.804157421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.278072916s FAIL (2025-10-06 22:19:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== :10.893937348 (320h14m56.356029831s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894116319 (320h14m56.356208812s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894442981 (320h14m56.356535464s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894517913 (320h14m56.356610386s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894828475 (320h14m56.356920948s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.894901423 (320h14m56.356993907s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895157502 (320h14m56.357249975s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895248334 (320h14m56.357340807s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895461119 (320h14m56.357553592s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895584485 (320h14m56.357676958s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895740581 (320h14m56.357833054s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.895970800 (320h14m56.358063273s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896095197 (320h14m56.35818768s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896436026 (320h14m56.358528499s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896447097 (320h14m56.35853958s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896471643 (320h14m56.358564126s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:10.896779309 (320h14m56.358871792s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Uploading piece... Oct 6 21:46:12.279095460 (320h14m57.741187944s) [upload=281874]: Next returned nil segment Oct 6 21:46:12.279171024 (320h14m57.741263517s) [upload=281874]: Waiting for error group managing segments... Oct 6 21:46:12.659282936 (320h14m58.12137541s) [noise=true , address=64.49.13.155:4923 , piece_id=R3ELJKAXYUNUDNNCVGDD3TSIPLRGBISHCBL6BRIMS7PTPZ3H7QSQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.745941981 (320h14m58.208034464s) [noise=true , address=145.79.227.41:4104 , piece_id=HZTHMQOHUAAJ7GOIQQVZGHDL7TURBLH7CWR4SMD6DAAMJFBZ6SXQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.757164832 (320h14m58.219257305s) [noise=true , address=145.79.227.27:4709 , piece_id=V2OIC66X6V775N77SEF6DXL5G2O23NTBXBKU3327DXOB2RQUD5QA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.759881254 (320h14m58.221973737s) [noise=true , address=145.79.227.22:4526 , piece_id=4G3YMBW7GYPYMYKJHAFXWBANMQ3ZYBPWEJXNYBBGMDEIPYVPO53Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:12.770054335 (320h14m58.232146808s) [noise=true , address=145.79.227.51:4429 , piece_id=RRVWEPAHC3PVI6E4AHIFLE3OBLMFRFLUZ2A46OGVSLXUFX6OUN2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.090820895 (320h14m58.552913378s) [noise=true , address=45.91.116.6:4993 , piece_id=5QLDYVFHUNQNDAUB36XNSRMKQMZOQMWVVUIOI7YLREH2UWTUNB5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.137972199 (320h14m58.600064682s) [noise=true , address=45.91.116.4:4958 , piece_id=ZXQW5UT3EMAH7E26XMHCPKDCAFTMI3V533E7OM5QZ46XGRAHLLCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.155198151 (320h14m58.617290624s) [noise=true , address=77.90.31.21:4881 , piece_id=UM7H6WJJNW2QHNN253Y3NUFPTNU7ZRTJRYPGXGMUBWIC4AUERNOA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.185240577 (320h14m58.64733306s) [noise=true , address=77.90.31.15:4716 , piece_id=VVWRPUZ4XB5H3GKWWVUBY2ULSPUUTWEA5FOBRBEY2SBNX6YLG2VQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.208195237 (320h14m58.67028772s) [noise=true , address=45.91.116.13:4185 , piece_id=GRLAPG6AUGWKTI7OL5N74GPL7PLX4367CHIRZUZ7BVTYDVGLRBLQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.210035230 (320h14m58.672127713s) [noise=true , address=45.91.116.10:4092 , piece_id=VMSNXZ4CHAA6DWBBGPNHNCL6SADAMEOFBTAYCSWFWBANVTQ5ROPA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.212687631 (320h14m58.674780104s) [noise=true , address=77.90.31.19:4847 , piece_id=W7T4AUGV5MIY7V7RU63HAHNOYFKRAVW2EYMLXW7W3JOLK6UCNRCQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.215839582 (320h14m58.677932065s) [noise=true , address=45.91.116.14:4214 , piece_id=NKFMCOSXTSTBTVGJHOTXILCVHF7TAESHIB456YEI32HCBUJZZF5A, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232182503 (320h14m58.694274986s) [noise=true , address=77.90.31.16:4758 , piece_id=TVIQCBKLSRJWTDWZ4WM5TEHWA7OR5WUMNZY2D6SOII2XZ57R4P2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: <nil> Oct 6 21:46:13.232217209 (320h14m58.694309692s) [seg_pos={0 0}, upload=281874]: Segment reached optimal threshold of 14 pieces. Oct 6 21:46:13.232357165 (320h14m58.694449648s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232366903 (320h14m58.694459386s) [noise=true , address=77.90.31.13:4659 , piece_id=3LMCEUOZ3DPQ7XSW5DSM5VWZYHPMYU2GVEGGYEDSPCEOISP7AIHQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232453358 (320h14m58.694545841s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wBqmJYPpcmfnt1zWABq5JyjU66NkLetiDvLMAaNTE1zh5bJgw): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232477885 (320h14m58.694570368s) [noise=true , address=77.90.31.18:4815 , piece_id=22FZWT53BFXPMZIEWMHE6JEZG5BLTZROQFRLRAOIXVFKH7LABE2Q, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232497182 (320h14m58.694589665s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1U16aJ2LZPaRGyRizyyCbMJjUgtuDcaHaMsrQFbXHuyVGRc1w7): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232508423 (320h14m58.694600906s) [noise=true , address=45.91.116.10:4107 , piece_id=KQC3RPR2SEA5UJICTPTOS2HA2GWIGTWOVGSIZNMIPF4PW53HEGAQ, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232588004 (320h14m58.694680487s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127RBQsp1SwQU6yWh6Y48QF3QCzeJxEGjob8gzF3LRutRZXA6A9): protocol: expected piece hash; context canceled; EOF Oct 6 21:46:13.232596701 (320h14m58.694689184s) [noise=true , address=45.91.116.5:4314 , piece_id=NJCYDZNYP7IP7Y24WLIZ5EEC67DQDPNKOSNAKWQ3ZKRFH2TY3NUA, seg_pos={0 0}, upload=281874]: Long tail cancellation due to optimal threshold Oct 6 21:46:13.232651124 (320h14m58.694743608s) [seg_pos={0 0}, upload=281874]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 6 21:46:13.232793385 (320h14m58.694885868s) [upload=281874]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 6 21:46:13.232800609 (320h14m58.694893092s) [upload=281874]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 6 21:46:13.565002502 (320h14m59.027094976s) [upload=281874]: Done uploading segments... Oct 6 21:46:13.565029424 (320h14m59.027121907s) [upload=281874]: Upload finished. err: <nil> Duration: 6.278106921s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.120824745s FAIL (2025-10-06 22:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.120855634s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.606555886s FAIL (2025-10-06 22:19:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.606589962s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)