Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 818154 reports (Page 12192 of 16364)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.098713171s FAIL (2025-10-06 22:54:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.09874417s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................XX...] duration: 6.156688616s FAIL (2025-10-06 22:54:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.156713332s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.199404522s FAIL (2025-10-06 22:53:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.199426864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.91619111s FAIL (2025-10-06 22:53:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.916257137s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 6.419174512s FAIL (2025-10-06 22:53:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.419243604s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.501139538s FAIL (2025-10-06 22:53:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.501169734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.308395922s FAIL (2025-10-06 22:53:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.308419237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 6.223396199s FAIL (2025-10-06 22:53:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.223425645s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.252529626s FAIL (2025-10-06 22:53:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 1s) [upload=282816]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:09.705226252 (321h20m55.167318735s) [upload=282816]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:09.963702680 (321h20m55.425795183s) [seg_pos={0 0}, upload=282816]: Begin upload segment... Oct 6 22:52:09.963769067 (321h20m55.42586155s) [seg_pos={0 0}, upload=282816]: Done begin upload segment. Oct 6 22:52:09.963805366 (321h20m55.425897839s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.964722172 (321h20m55.426814655s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965320972 (321h20m55.427413455s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.965861641 (321h20m55.427954124s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966318392 (321h20m55.428410875s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.966743421 (321h20m55.428835914s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967180904 (321h20m55.429273387s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.967637232 (321h20m55.429729715s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968071450 (321h20m55.430163933s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.968551394 (321h20m55.430643877s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.970344448 (321h20m55.432436921s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.972779935 (321h20m55.434872418s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.975138947 (321h20m55.43723142s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.977040068 (321h20m55.439132551s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.979185262 (321h20m55.441277745s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.980790829 (321h20m55.442883302s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.981942793 (321h20m55.444035276s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:09.983640946 (321h20m55.44573343s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Uploading piece... Oct 6 22:52:11.327777738 (321h20m56.789870221s) [upload=282816]: Next returned nil segment Oct 6 22:52:11.327802957 (321h20m56.78989544s) [upload=282816]: Waiting for error group managing segments... Oct 6 22:52:11.701918913 (321h20m57.164011396s) [noise=true , address=64.49.13.139:4472 , piece_id=F2MSY2653T6HCNVVT6Q4QTCRBWNFJ72B5BP72T53WBPMUO3T4OFQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.834055432 (321h20m57.296147915s) [noise=true , address=145.79.227.30:4786 , piece_id=CH3JD5B75Q4BZCCFT5SSQAZQVTMT6IFMZ7BK33ZMOWGQNTA6J6FQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.841313904 (321h20m57.303406387s) [noise=true , address=145.79.227.28:4745 , piece_id=3G62UTZWOTII2S4IPLXU27JZ5RT3S7YJWID7C6GYIOCHS5YR2ZUQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.857094161 (321h20m57.319186644s) [noise=true , address=145.79.227.52:4469 , piece_id=2MTFTISBR5X6FQTDF7WWYJFUNEXU324X47KIL3NI7AC4PGDQ47YA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.860301359 (321h20m57.322393832s) [noise=true , address=145.79.227.32:4859 , piece_id=NKYPYVTQCV2AIVFA5CZ5SD26TYHBHATKATHSDU4ZEMODDV3K7RAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:11.875424816 (321h20m57.337517299s) [noise=true , address=145.79.227.25:4631 , piece_id=XUS7W2CMNH5BLFKWSWA5G2COFLCUUTLOMKORJQDKT3EUC3WDSI5A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:12.141430363 (321h20m57.603522846s) [noise=true , address=145.79.227.22:4522 , piece_id=Q5QGDIDCEVMHYFBWSET264LRQJ76537LYCUEABNXJYSTXFSKJIAQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:58.888021309 (321h21m44.350113792s) [noise=true , address=77.90.31.10:4544 , piece_id=HT3OBHXMT75YHTQEAZNL4GNDKRHIG3WYV22MR6QCEMYSHLCNJY2A, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.122392930 (321h21m44.584485403s) [noise=true , address=77.90.31.21:4881 , piece_id=O44FZJVXJEFTJEYE4AT4IAGDW5W5WQPFBVDSD3XDHM6DXZ6LJOVA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:52:59.569730452 (321h21m45.031822935s) [noise=true , address=45.91.116.9:4264 , piece_id=E5AF6NY5Y664WLCSEX3CNXXCPQSLKBNMQ2QK273RWIOFOKPBVUCQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:00.216736676 (321h21m45.678829159s) [noise=true , address=45.91.116.8:4061 , piece_id=HURRWATHSQZ6UGZWO4JZ6GRM3GLUEVX4Q2VVR524PQG5IQMZLVQA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:01.785256907 (321h21m47.24734939s) [noise=true , address=45.91.116.11:4128 , piece_id=EAPS4LEZB4CIWFXSZQYEJGHB4IWMQK5GKZFDYPYPELPMBMZYALRQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:02.677904791 (321h21m48.139997274s) [noise=true , address=77.90.31.11:4583 , piece_id=KGYVBYDFPX3ZG6XGG3IHUBIUVJ3VTDUPM5Z4BGSA6T7KWCMINBGQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: <nil> Oct 6 22:53:08.980289045 (321h21m54.442381538s) [noise=true , address=45.91.116.13:4173 , piece_id=6MGJHHI64O4HXFZBYD753RDHPIJTZ3P6BXKIPORHQN5GKYDDYA2Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1J3HZY416Dm9F5zRXtoDdt8s8tGoU6GpkwS14CXJUzd3zpcYa1, address:45.91.116.13:4173): protocol: write tcp 188.166.225.170:38752->45.91.116.13:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980437188 (321h21m54.442529671s) [noise=true , address=77.90.31.15:4726 , piece_id=E5Q2766SG4G7FEKLJ6SK6VR7II5732CC5CEEBBNBKD5MGEMIFATQ, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:12kMaukCkCJSc3ukAZu3thLgzXGGzNq44dGyEjCPqoY3PSYVEm3, address:77.90.31.15:4726): protocol: read tcp 188.166.225.170:54666->77.90.31.15:4726: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980463948 (321h21m54.442556431s) [noise=true , address=77.90.31.15:4729 , piece_id=LS2OM764QSG2OIXGGUOFJPGARCWWFFZSEH3CVIRH77LLATQ6DDRA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1ArQdwHzHSm7c2QZkTt1kQsfzocvCNG2D9yoQgnmbS7HYLCXhy, address:77.90.31.15:4729): protocol: read tcp 188.166.225.170:37314->77.90.31.15:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980541937 (321h21m54.44263442s) [noise=true , address=45.91.116.13:4162 , piece_id=4UJQUI4E55U3E4AKXA3QN2B57ZJTEBLCZ2XL5OXSCYI6H37INAEA, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1geUSD7qjW2yuMayTJ1Zy1KTYVC19adZ9nUWMRtmg2TktJSsm4, address:45.91.116.13:4162): protocol: read tcp 188.166.225.170:45390->45.91.116.13:4162: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980615156 (321h21m54.442707639s) [noise=true , address=77.90.31.18:4808 , piece_id=52DZLDHEUIB4354GGWUQ3D5EZUPZCB56EPDOY2JUMNQVWVQ2S46Q, seg_pos={0 0}, upload=282816]: Done uploading piece. err: ecclient: upload failed (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, address:77.90.31.18:4808): protocol: read tcp 188.166.225.170:56916->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:53:08.980647698 (321h21m54.442740182s) [seg_pos={0 0}, upload=282816]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Oct 6 22:53:09.230792423 (321h21m54.692884967s) [upload=282816]: Done uploading segments... Oct 6 22:53:09.230828702 (321h21m54.692921185s) [upload=282816]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.252557871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 14.667218052s FAIL (2025-10-06 22:53:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 14.66724352s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.000758504s FAIL (2025-10-06 22:52:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000775056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.248909579s FAIL (2025-10-06 22:52:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Oct 6 22:52:03.142797923 (321h20m48.604890407s) [upload=282815]: Starting upload Oct 6 22:52:03.142882876 (321h20m48.604975359s) [upload=282815]: Done starting upload Oct 6 22:52:03.143119175 (321h20m48.605211658s) [upload=282815]: Uploading segments... Oct 6 22:52:03.143181053 (321h20m48.605273536s) [upload=282815]: Got next segment. Inline: false Oct 6 22:52:03.143225828 (321h20m48.605318321s) [upload=282815]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:52:03.143231440 (321h20m48.605323923s) [upload=282815]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:52:03.394372661 (321h20m48.856465134s) [seg_pos={0 0}, upload=282815]: Begin upload segment... Oct 6 22:52:03.394454848 (321h20m48.856547332s) [seg_pos={0 0}, upload=282815]: Done begin upload segment. Oct 6 22:52:03.394491658 (321h20m48.856584132s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.394590026 (321h20m48.856682509s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395251425 (321h20m48.857343908s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.395728183 (321h20m48.857820666s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396142962 (321h20m48.858235445s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.396535290 (321h20m48.858627773s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397107750 (321h20m48.859200233s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.397540904 (321h20m48.859633387s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.398063599 (321h20m48.860156083s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400392765 (321h20m48.862485248s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.400471505 (321h20m48.862563988s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.402567045 (321h20m48.864659529s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.406381188 (321h20m48.868473711s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.408318086 (321h20m48.870410569s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.410257660 (321h20m48.872350143s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:03.412078567 (321h20m48.87417105s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Uploading piece... Oct 6 22:52:05.092482443 (321h20m50.554574926s) [upload=282815]: Next returned nil segment Oct 6 22:52:05.092504556 (321h20m50.554597029s) [upload=282815]: Waiting for error group managing segments... Oct 6 22:52:05.374233372 (321h20m50.836325855s) [noise=true , address=145.79.227.14:4837 , piece_id=5PJOKFPZI2OLREPSUCMZOHD7I4G5JIYCLE4TOCMU7A4RUV33LCLQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.374719768 (321h20m50.836812251s) [noise=true , address=145.79.227.4:4187 , piece_id=MRNUYEWMV6TYAEHLNGAI4VZDRWG7EX5ZVZQGBWK7BMAKMVNMY35A, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.393906613 (321h20m50.855999106s) [noise=true , address=145.79.227.3:4151 , piece_id=SKPCCRFXUR6NVNWICJT5N3L6YEB4ZTSYZB4ILMIGWTAFZT3DHMUA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.394673884 (321h20m50.856766367s) [noise=true , address=145.79.227.11:4730 , piece_id=2JN4BHGUP5CEC3UQOGPNVK7NXOXFMCKB6OK2XEY7U54RXI63GDNA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.396965227 (321h20m50.8590577s) [noise=true , address=145.79.227.16:4909 , piece_id=3G3PWXNDJFZXJTZU56XEXBMXZGX3MAYE5D7AJTO6TT6SRXJZNTIQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.399932107 (321h20m50.86202459s) [noise=true , address=145.79.227.2:4113 , piece_id=KMT523MH6PNE2ELVYMBH5OLTA4KPOWQ6DBE5GQ2WHPAB6T3K2WHQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401058180 (321h20m50.863150653s) [noise=true , address=145.79.227.6:4258 , piece_id=57D23RXUMGNMQW2J3QGSXDJDW66YVXBRE7SNTNJRIR746T2XWLKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.401559966 (321h20m50.86365244s) [noise=true , address=145.79.227.18:4007 , piece_id=4ZLTAOLZORQXO3MPM2IN7S5E3EHS5HFXWJTQGZYGQRP5DCEIQ3ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.403428885 (321h20m50.865521358s) [noise=true , address=145.79.227.18:4006 , piece_id=7HH4J3PLBLHBS5JQAKXGQXYL4FALWRABAYS3L4RFGZKYCGTPCWKA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:05.436020524 (321h20m50.898113007s) [noise=true , address=145.79.227.19:4079 , piece_id=4MPKS54YF2QAKOKZKHFX4FDZFXWRVW2D4JTCAJITJSBP3HXHP7ZQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: <nil> Oct 6 22:52:32.424567253 (321h21m17.886659746s) [noise=true , address=45.91.116.14:4190 , piece_id=PYEXICOSZPOFURPJPDYPV3KUTQMXHKXZ5LWOJ7VGPWEQ57S6V3AA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424745832 (321h21m17.886838306s) [noise=true , address=45.91.116.9:4263 , piece_id=ZWC4IXBGKFVROM456M3WSNU4UNUUPUXUUQTAYFZ6A465EQMOVIGA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: read tcp 188.166.225.170:48620->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424765971 (321h21m17.886858444s) [noise=true , address=77.90.31.10:4519 , piece_id=5MKRVSGTTOYUGWJNVTXUZFGFMQVSH2KSXJKHTCYWY455LLB4H2HQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: read tcp 188.166.225.170:55108->77.90.31.10:4519: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.424885088 (321h21m17.886977561s) [noise=true , address=45.91.116.8:4047 , piece_id=DU4HIURHOJGJEH56O4H2LRRHGDS2ANZWS262FWIGWORROMUWPVBQ, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: read tcp 188.166.225.170:33706->45.91.116.8:4047: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425031366 (321h21m17.887123839s) [noise=true , address=45.91.116.6:4975 , piece_id=3O6USQV5CSC3PZ2QZTHBUUQB4RDTA5VYDQ2PE2MOGPB5TISRNW4Q, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: read tcp 188.166.225.170:58100->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425046375 (321h21m17.887138858s) [noise=true , address=77.90.31.15:4698 , piece_id=DOJVVP6YWMYWSKTPUPMG3JNM622IEKXFD7YCUDFL27MPUALFZRTA, seg_pos={0 0}, upload=282815]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: read tcp 188.166.225.170:40278->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:52:32.425075130 (321h21m17.887167603s) [seg_pos={0 0}, upload=282815]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Oct 6 22:52:32.670458720 (321h21m18.132551203s) [upload=282815]: Done uploading segments... Oct 6 22:52:32.670493546 (321h21m18.132586029s) [upload=282815]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.248938483s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000351898s FAIL (2025-10-06 22:52:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000374019s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.252728963s FAIL (2025-10-06 22:48:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.252783857s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.187818644s FAIL (2025-10-06 22:47:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.187846347s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.148690628s FAIL (2025-10-06 22:46:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.1487337s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.058572075s FAIL (2025-10-06 22:45:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.058597985s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.627331073s FAIL (2025-10-06 22:45:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.627367944s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.718497702s FAIL (2025-10-06 22:44:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.718525002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.517643159s FAIL (2025-10-06 22:44:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.517670169s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.852742996s FAIL (2025-10-06 22:43:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.852772714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.427779953s FAIL (2025-10-06 22:43:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.427812135s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.181279722s FAIL (2025-10-06 22:43:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.181305763s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.469637457s FAIL (2025-10-06 22:42:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.469758757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.23946258s FAIL (2025-10-06 22:41:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.23948847s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.091602993s FAIL (2025-10-06 22:41:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.091634131s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.438442086s FAIL (2025-10-06 22:41:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.438474788s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.345739493s FAIL (2025-10-06 22:41:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.345770752s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.129574828s FAIL (2025-10-06 22:40:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.129606179s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.198792611s FAIL (2025-10-06 22:40:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.198821737s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.84029064s FAIL (2025-10-06 22:40:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.840320477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.595741122s FAIL (2025-10-06 22:39:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.595763264s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.296780466s FAIL (2025-10-06 22:39:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.296864667s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.301761244s FAIL (2025-10-06 22:39:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.301791501s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.398910301s FAIL (2025-10-06 22:38:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.398934596s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.488808453s FAIL (2025-10-06 22:38:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.488842187s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.16507717s FAIL (2025-10-06 22:38:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.165104322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.76834435s FAIL (2025-10-06 22:38:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.768378685s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.209529065s FAIL (2025-10-06 22:37:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.20956294s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.292283896s FAIL (2025-10-06 22:37:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.292313152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.574290756s FAIL (2025-10-06 22:37:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.574321564s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.218705197s FAIL (2025-10-06 22:37:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.218750412s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.056206671s FAIL (2025-10-06 22:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.056234213s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.474388194s FAIL (2025-10-06 22:36:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.474422979s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.158991932s FAIL (2025-10-06 22:35:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.159017562s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.35009687s FAIL (2025-10-06 22:35:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.350139421s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.375288612s FAIL (2025-10-06 22:34:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.37531397s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.508312089s FAIL (2025-10-06 22:34:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.50841785s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.363017854s FAIL (2025-10-06 22:34:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.363046908s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.702533683s FAIL (2025-10-06 22:33:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Oct 6 22:04:19.954155862 (320h33m5.416248335s) [upload=282127]: Starting upload Oct 6 22:04:19.954229041 (320h33m5.416321514s) [upload=282127]: Done starting upload Oct 6 22:04:19.954335063 (320h33m5.416427536s) [upload=282127]: Uploading segments... Oct 6 22:04:19.954401840 (320h33m5.416494313s) [upload=282127]: Got next segment. Inline: false Oct 6 22:04:19.954429042 (320h33m5.416521515s) [upload=282127]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 6 22:04:19.954433932 (320h33m5.416526415s) [upload=282127]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 6 22:04:20.206252742 (320h33m5.668345215s) [seg_pos={0 0}, upload=282127]: Begin upload segment... Oct 6 22:04:20.206308428 (320h33m5.668400901s) [seg_pos={0 0}, upload=282127]: Done begin upload segment. Oct 6 22:04:20.206335209 (320h33m5.668427682s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206351429 (320h33m5.668443912s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.206978754 (320h33m5.669071237s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207268326 (320h33m5.669360799s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207438720 (320h33m5.669531203s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207556955 (320h33m5.669649438s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207918523 (320h33m5.670011006s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.207933382 (320h33m5.670025865s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208301963 (320h33m5.670394446s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208384471 (320h33m5.670476954s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208571146 (320h33m5.670663619s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208976679 (320h33m5.671069162s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.208991907 (320h33m5.67108438s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.209479065 (320h33m5.671571558s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210112461 (320h33m5.672204934s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:20.210787928 (320h33m5.672880411s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Uploading piece... Oct 6 22:04:21.905295742 (320h33m7.367388225s) [upload=282127]: Next returned nil segment Oct 6 22:04:21.905319687 (320h33m7.36741217s) [upload=282127]: Waiting for error group managing segments... Oct 6 22:04:22.187217893 (320h33m7.649310376s) [noise=true , address=145.79.227.7:4295 , piece_id=PE3VUAE3KYUG6NRLCCWEBZU2KLVSZGCP3FWWQUXYUIZL4UGRGWZA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.188571641 (320h33m7.650664134s) [noise=true , address=145.79.227.4:4185 , piece_id=X3YDM74366ZZJ4LBPWR7JPZYWQ7PGA2SRMYXAELKFACSJIHZFFDQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.189097572 (320h33m7.651190055s) [noise=true , address=145.79.227.17:4969 , piece_id=IFDDGPQKLWS64FOAN224GQ2GGQRXBPC3ZZJ5YCL45HMPFH37CNVQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.191981073 (320h33m7.654073556s) [noise=true , address=145.79.227.2:4114 , piece_id=OJZMSPV4JGL3K6JAEHOSZLYD2UHU2DQFMQMIX7LX5AZWY4VNMXYQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192460917 (320h33m7.6545534s) [noise=true , address=145.79.227.5:4221 , piece_id=M4D53WIW56NSULTECTIIGXUPTX5PW5KNPIN66EH3PVGA36IWMDFQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.192946201 (320h33m7.655038684s) [noise=true , address=145.79.227.19:4078 , piece_id=PWJ2UTJLKJ64PRZ6NSOVWNTO7DNQ3UYRI7UQVAQN3YFTBXGOPJNA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.194315208 (320h33m7.656407681s) [noise=true , address=145.79.227.16:4910 , piece_id=XGPGRFFEUUS3G54L2SXFO5AJXZAR6NXZORG36IPOOAMBONX42AUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.196870804 (320h33m7.658963288s) [noise=true , address=145.79.227.11:4730 , piece_id=AAYIDTHT23IF4YWVKHZ6QKTRBTPL3RQ4JMRSAZI64RWIYVRGQAKQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.200828571 (320h33m7.662921054s) [noise=true , address=145.79.227.18:4006 , piece_id=2KULBJLDJ53TZKKEUH4OW3LWM4FNMJJDQKW4S2LOLH2D5N554K2Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207679346 (320h33m7.669771819s) [noise=true , address=145.79.227.13:4801 , piece_id=S7YOESXL2Y3ZJCJ6ZU7DFBUWBUZMO4RDCUZILBPBKHNMVC2BOIMA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.207907851 (320h33m7.670000334s) [noise=true , address=145.79.227.12:4767 , piece_id=PZQJAN2JXM2CFISYECWCQQLDD3PEVLL53QFATQXMWE7ON6TNPRUA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:22.212970191 (320h33m7.675062674s) [noise=true , address=145.79.227.14:4838 , piece_id=QEM66UEXQ7V5QMVO5RGO232HUK6AF3W2L5GXVEBPJ4KJ65IWMYWQ, seg_pos={0 0}, upload=282127]: Done uploading piece. err: <nil> Oct 6 22:04:49.261098871 (320h33m34.723191364s) [noise=true , address=45.91.116.7:4009 , piece_id=D5K5KQ5UMOX7F3ELEATZ6QSONG42DI6B2AWXCHFGV4NQKKZLGNRA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:51696->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261706808 (320h33m34.723799281s) [noise=true , address=77.90.31.21:4877 , piece_id=YK6MFFP6XZ7EALDKGHHPSQ4OCKJAAR6EGG7EVMYQATVK22B4XCVA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:47292->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.261784075 (320h33m34.723876558s) [noise=true , address=45.91.116.6:4974 , piece_id=PERSGTTG2ZGFB56JNSQDULE2KAMVCPABKBIKOOONWMC5A4XHUXDA, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: read tcp 188.166.225.170:33956->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262060922 (320h33m34.724153405s) [noise=true , address=45.91.116.9:4262 , piece_id=AGVES5DE2LJSFWMR3YEKIIVII4L5HYPZ6MYK5F3AENK43ADJIX7Q, seg_pos={0 0}, upload=282127]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:54010->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Oct 6 22:04:49.262095908 (320h33m34.724188392s) [seg_pos={0 0}, upload=282127]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Oct 6 22:04:49.500322617 (320h33m34.96241509s) [upload=282127]: Done uploading segments... Oct 6 22:04:49.500345650 (320h33m34.962438133s) [upload=282127]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.702567818s core uplink us-select-1

1
...
12190
12191
12192
12193
12194
...
16364